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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  in  partial 
response  to  the  task  entitled  “Assessment  of  Traditional  and  Emerging  Approaches  to  the 
Detection  and  Identification  of  Surface  and  Buried  Unexploded  Ordnance.” 
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SUMMARY 


A.  BACKGROUND 

The  Environmental  Security  Technology  Certification  Program  (ESTCP) 
sponsored  the  development  of  the  Multi-sensor  Towed  Array  Detection  System 
(MTADS)  at  the  Naval  Research  Laboratory  (NRL).  This  document  analyzes  the  results 
of  a  test  of  MTADS  at  Jefferson  Proving  Ground  (JPG)  from  14-24  January  1997. 

MTADS  is  a  vehicle- towed  sensor  with  an  array  of  coils  that  can  be  used  as  a 
magnetometer  (MAG)  or  an  electromagnetic  induction  (EMI)  sensor.  There  were  three 
scenarios  in  the  MTADS  test,  each  roughly  10  acres  in  size.  It  took  5  to  8  hours  of  actual 
survey  time  to  cover  most  of  the  10  acres,  which  as  a  practical  matter  meant  that  a  day  or 
two  was  needed  to  survey  each  scenario  with  each  sensor  configuration  (MAG  or  EMI). 

Known  baseline  ordnance  appropriate  to  the  scenario  were  emplaced  at  random 
locations  on  the  site.  Table  S-l  shows  the  number  of  emplaced  ordnance  in  the  areas  that 
were  surveyed,  as  well  as  the  number  of  declarations  that  the  MTADS  operators  believed 
were  ordnance.  The  probability  of  detection,  PD,  also  shown  in  Table  S-l,  comes  from  the 
number  of  those  declarations  that  were  correct. 


Table  S-1 .  Detection 


Scenario 

Number  of 
Baseline 
Ordnance 

Number  of 
Ordnance 
Declarations 

Number  of 
Correct 
Ordnance 
Declarations 

PD(%) 

1 

Aerial  Gunnery 

47 

185 

45 

95.7 

2 

Artillery  and  Mortars 

73 

216 

70 

95.9 

3 

Submunitions  and 
Grenades 

86 

215 

80 

93 

The  number  of  false  alarms  (FAs)  is  defined  as  the  number  of  declarations  as 
ordnance  that  did  not  correspond  to  baseline  ordnance.  The  false-alarm  rate  is  shown  in 
Table  S-2. 
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Table  S-2.  False  Alarm  Rates 


Scenario 

Area  Surveyed 
(Square  Meters) 

Number  of  False 
Alarms 

False  Alarm  Rate 
(per  Hectare)8 

1  Aerial  Gunnery 

33,445  (8.265 
acres) 

140 

41.9 

2.  Artillery  and 
Mortars 

146 

37.6 

3.  Submunitions 
and  Grenades 

29,729  (7.347 
acres) 

135 

45.4 

*  A  hectare  is  10,000  m2  or  2.47  acres. 

The  addition  of  EMI  sensor  information  was  crucial  for  reducing  the  FAR, 
particularly  in  scenarios  1  and  3,  where  it  took  part  in  every  good  detection.  In  scenario  1 
it  was  not,  by  itself,  responsible  for  any  false  alarms.1  Table  S-3  shows  the  discriminants 
that  were  derived  in  each  case  and  the  potential  FAR  reduction. 

The  obvious  implications  are  that  between  25-  and  50-percent  reduction  in  false 
alarms  should  be  achievable  with  fairly  rudimentary  discrimination  algorithms.  An 
important  caveat  to  this  finding  is  that  it  is  essential  to  know  what  kind  of  scenario  one  is 
in.  No  useful  discriminant  was  found  that  was  good  in  all  three  scenarios.  Also,  it  should 
be  emphasized  that  although  Table  S-3  accurately  shows  the  FAR  reduction  that  would 
have  been  achieved  in  the  JPG  MTADS  test,  it  is  not  necessarily  indicative  of 
performance  in  other  tests.  Specifically,  the  discriminant  thresholds  chosen  were  such  as 
to  allow  zero  missed  detections  on  account  of  the  false-alarm  mitigation.  The  real 
population  distribution  may  have  tails  that  would  imply  some  number  of  missed 
detections  as  the  tradeoff  for  reducing  the  FAR.  Only  further  tests  or  a  very  good 
understanding  of  the  detection  physics  can  determine  this.  On  the  other  hand,  knowledge 
of  the  detection  physics  could  very  well  allow  one  to  do  false  alarm  mitigation  better  than 

indicated  in  Table  S-3. 


1  This  assumes  that  those  declarations  for  which  there  were  both  EMI  and  MAG  data  werenot  more 
difficult  (in  some  sense)  fhan  those  for  which  only  MAG  or  only  EMI  data  were  available.  That  is,  no 
essential  synergy  is  assumed  between  MAG  and  EMI. 


Table  S-3.  Potential  FAR  Reduction  by  Using  Linear  Discriminants 


Legend:  ZMAG  =  fitted  depth,  magnetometer 
INCLIN  =  fitted  inclination 
RFE_EM  =  ferrous  radial  size,  EMI  sensor 
GOF_EM  =  goodness  of  fit,  EMI  sensor 
RMAG  =  fitted  radius,  magnetometer 
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I.  INTRODUCTION 


In  the  past  several  years  the  Environmental  Security  Technology  Certification 
Program  (ESTCP)  has  sponsored  development  of  an  unexploded  ordnance  (UXO) 
detection  system  mounted  on  a  vehicle.  The  vehicle  systematically  traverses  a  given 
terrain  region  and  produces  data  in  or  nearly  in  real  time  that  indicate  the  existence  and 
location  of  UXO.  This  system  was  developed  at  the  Naval  Research  Laboratory  (NRL) 
and  is  called  the  Multi-Sensor  Towed  Array  Detection  System  (MTADS). 

MTADS  has  been  through  three  previous  tests.  The  first  was  a  technical 
evaluation  (TECHEVAL)  to  verify  system  technical  characteristics.  The  second  was  a 
demonstration  at  the  Magnetic  Test  Range  (MTR)  in  Twentynine  Palms,  California,  in 
December  1996.  The  third  was  a  commercial  demonstration  in  the  third  Jefferson  Proving 
Ground  test  (JPG  III). 

The  test  that  is  the  subject  of  this  document  was  a  demonstration  of  the  MTADS 
at  Jefferson  Proving  Ground  (JPG)  conducted  from  14-24  January  1997.  The  test  was 
conducted  in  accordance  with  the  Technology  Demonstration  Plan  (TDP)  (Ref.  1-1)  dated 
20  December  1996. 

This  document  assesses  the  overall  detection  performance  of  the  MTADS  at  JPG 
during  this  test.  Supporting  analysis,  as  well  as  data  providing  insight  into  MTADS 
operation,  is  also  provided.  The  structure  of  this  document  is  straightforward.  After  an 
introduction  to  the  MTADS  system  and  the  test,  an  analysis  of  the  data  is  presented.  A 
final  chapter  is  devoted  to  conclusions  and  directions  for  further  study. 

A.  THE  MTADS 

The  MTADS  is  an  array  of  sensors  on  two  carts  and  the  low-magnetic-signature 
vehicle  that  tows  them.  The  sensor  arrays  are  either  of  magnetometers  or  electromagnetic 
induction  (EMI)  coils.  The  magnetometers  can  also  be  used  as  a  magnetic  gradiometer,  a 
configuration  that  provides  data  with  much  the  same  information  as  the  magnetometer. 
Only  the  magnetometer  array  was  used  in  the  test  analyzed  in  this  document. 

MTADS  is  designed  to  minimize  the  noise  and  signal  bias  caused  by  the  tow 
vehicle  and  carts.  Good  navigation,  essential  to  quality  data  collection,  has  been  a 
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principal  issue,  both  in  terms  of  potential  remediation  protocol  (that  is,  finding  the 
ordnance  when  digging  for  it)  and  in  the  conduct  of  tests.  The  vehicle  and  sensor 
positions  are  determined  by  a  real-time  kinematic  differential  global  positioning  system 
(RTK-DGPS),  providing  location  accuracy  to  within  5  cm.  This  high  location  accuracy 
allows  the  co-registration  and  integration  of  data  taken  at  different  times. 

MTADS  includes  a  data  acquisition  system  (MTADS-DAQ)  and  a  data  analysis 
system  (MTADS-DAS).  MTADS-DAQ  is  software  used  for  collection  and  saving  of 
magnetometer  or  EMI  data  during  MTADS  operation.  MTADS-DAS  is  a  stand-alone 
suite  of  programs  used  for  later  data  analysis.  Based  on  Interactive  Data  Language  (IDL) 
tools,  MTADS-DAS  runs  on  a  Unix  workstation.  It  is  a  menu-driven  system  that  permits 
easy  visualization  of  the  data  and  allows  the  user  to  hand  pick  anomalies  within  the 
sensor  data  and  then  determine  features  or  parameters  associated  with  the  anomaly.  These 
parameters  are  used  for  later  remediation  of  suspected  UXO.  The  parameter  set  depends 
on  the  sensor  being  towed.  For  the  magnetometer,  the  parameters  are 

•  Location  (Northing  and  Easting  coordinates) 

•  Depth 

•  Size 

•  Magnetic  Moment 

•  Azimuth  (of  dipole  moment) 

•  Inclination  (of  dipole  moment) 

•  GoF. 

For  the  EMI,  the  parameters  are 

•  Location  (Northing  and  Easting  coordinates) 

•  Depth 

•  Ferrous  Size 

•  Nonferrous  Size 

•  GoF. 

B.  THE  JPG  FIELD  TEST 

From  14-24  January  1997,  an  NRL  team  surveyed  an  area  with  MTADS. 
The  location  was  JPG,  north  of  Madison,  Indiana  (Figure  1-1).  There  are  two  principal 
areas  at  JPG  that  are  used  to  test  means  of  UXO  detection.  The  “North  Site”  was  the 
scene  of  the  MTADS  test  (Figure  1-2). 
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Figure  1-1.  Jefferson  Proving  Ground  North  of  Madison,  Indiana 

The  JPG  North  site  has  an  area  of  16  hectares  (16,000  m2),  or  approximately 
40  acres.  It  is  1,320  ft  on  a  side  and  is  divided  into  quadrants,  each  roughly  10  acres. 
Each  quadrant  corresponds  to  a  specific  scenario  and  is  further  divided  as  evenly  as 
possible  into  100-ft  (30.48-m)  squares.  Some  “leftover”  cells  at  the  southern  and  western 
borders  of  the  test  site  are  only  20  ft  wide. 

Three  of  the  quadrants  were  used  for  the  MTADS  test  analyzed  in  this  document. 
The  first  scenario  entailed  detection  of  ordnance  from  aerial  gunnery  (southeast 
quadrant),  the  second  from  artillery  and  mortars  (northwest  quadrant),  and  the  third  from 
submunitions  and  grenades  (southwest  quadrant). 

The  NRL  team  deployed  the  MTADS  vehicle  and  sensor  to  the  test  site,  took 
1  day  to  set  up,  and  began  to  survey  the  site  on  15  January  1997.  Table  1-1  gives  the 
survey  details.  They  estimated  (Ref.  1-2)  that  they  would  cover  about  10  acres  per  day, 
assuming  a  collection  rate  of  2  m/s.  This  would  imply  1  day  per  scenario.  The 
information  in  Table  1-1  (from  Ref.  1-2)  indicates  that  this  was  attained  in  terms  of  hours 


1-3 


Figure  1-2.  North  Test  Site  at  Jefferson  Proving  Ground 
Table  1-1.  Survey  Dates  and  Times.  After  Ref.  1-2. 


Scenario 

MAG  Survey 
Dates 

EMI  Survey 
Dates 

MAG  Survey 
Time  [Hours] 

EMI  Survey  Time 
[Hours] 

Aerial  Gunnery 

15-16  January 
1997 

22  January  1 997 

5.3 

5.8 

Artillery  and 
Mortar 

16-17  January 
1997 

17-18  January 
1997 

5.4 

7.8 

Grenades  and 
Submunitions 

21  January  1997 

18-19  January 
1997 

5.9 

7.8 

actually  spent  surveying.  The  extra  time  in  terms  of  days  (say  a  factor  of  two)  would  need 
to  be  incorporated  for  actual  survey  planning.  Certainly  this  is  also  dependent  on  terrain 
and  weather.  The  weather  in  this  test  was  cold  (January  in  Indiana),  having  periods  with 
wind  chills  below  -20  °F  (Ref.  1-2).  In  addition,  not  all  10  acres  in  each  scenario  were 
actually  surveyed.  Actual  survey  coverage  is  examined  more  closely  later  in  this  section. 
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NRL  personnel  used  the  given  characteristics  of  the  scenario  to  help  in  the 
classification  of  declarations  as  ordnance  or  nonordnance.  They  were  specifically  told  the 
smallest  size  of  ordnance  in  each  scenario  (Ref.  1-2,  Section  3.1).  In  addition,  NRL 
appears  to  have  used  the  scenario  type  to  exclude  targets  that  were  too  large.  This 
occurred  in  the  last  scenario  (submunitions  and  grenades)  when  a  signal  apparently  like 
that  of  a  “large  bomb”  was  excluded  as  “nonordnance.” 

Only  a  portion  of  the  test  site  was  actually  surveyed.  Surveyed  grid  cells  are 
indicated  in  Table  1-2,  based  on  information  in  Reference  1-2.  The  corresponding 
surveyed  areas  appear  in  Table  1-3.  MTADS  surveyed  83.5  percent  of  the  approximately 
30-acre  area.  Various  reasons  account  for  not  surveying  a  region.  For  instance,  a  tree  line 
might  have  interfered  with  GPS  or  the  terrain  might  have  been  too  treacherous.  Whatever 
the  cause,  only  those  grid  cells  specified  by  NRL  as  surveyed  in  Reference  1-2  are 
considered  further  in  this  document. 

Table  1-2.  Surveyed  Grid  Cells.  S  indicates  the  cell  was  surveyed.  Red  indicates 
the  area  for  scenario  1 ,  blue  for  scenario  2,  and  green  for  scenario  3. 

The  width  of  grid  cells  in  the  edges  labeled  N  or  1 4  is  20  ft. 

Thus,  grid  cell  N14  is  20  ft  x  20  ft. 


1 

2 

3 

4 

5 

6 

7 

8 
9_ 
10 
11 
12 

13 

14 


Table  1-3.  Areas  of  Surveyed  Cells 


Scenario 

Ordnance 

Number  of  Cells 

Area  [square  meters] 

Proportion 

Surveyed 

[%] 

Total 

Surveyed 

Total 

Surveyed 

1 

Aerial 

Gunnery 

36  +  6(.2) 

=  37.2 

35  +  5(.2) 

=  36 

34,560 
(8.541  acres) 

33,445 
(8.265  acres) 

96.8 

2 

Artillery  and 
Mortar 

49  +  7(.2) 

=  50.4 

42  +  2(.2) 

=  42.4 

46,823 

(11.571 

acres) 

39,391 
(9.735  acres) 

84.1 

3 

Grenades  and 
Submunitions 

42  + 1 3(.2) 

+  (.2)(.2) 

=  44.64 

31  +  5(.2) 

=  32 

29,729 
(7.347  acres) 

29,729 
(7.347  acres) 

71.7 

All 

132.24 

110.4 

122,855 

(30.361 

acres) 

102,565 

(25.347 

acres) 

83.5 

NRL  provided  three  files  of  UXO  declarations  to  IDA  for  evaluation,  one  file  for 
each  scenario.  Some  of  the  declarations  were  in  cells  that  were  not  indicated  as  being 
surveyed.  To  complicate  matters  further,  our  calculation  of  which  declarations  belonged 
in  which  cells  differed  in  some  instances  from  that  indicated  by  NRL  in  the  declaration 
files.  That  is,  the  X-Y  coordinates  in  the  file  did  not  correspond  to  the  grid  cell  indicated 
in  the  file.  Table  1-4  shows  the  disagreements.  The  grid  and  object  numbers  assigned  to 
each  declaration  by  NRL  uniquely  identify  it.  Of  the  271  declarations  in  scenario  1,  there 
were  15  differences;  of  283  in  scenario  2, 17;  and  of  the  242  in  scenario  3,  6. 

Most  of  the  disagreements  were  on  declarations  that  were  very  close  to  the  grid 
cell  boundary,  usually  within  1  m.  Distances  from  the  recorded  X— Y  positions  to  the 
NRL-assigned  grid  cell  are  also  shown  in  Table  1-4.  Four  declarations  in  M2  were  more 
than  9  m  from  the  NRL-assigned  cell  of  N2.  These  were  the  largest  differences  except  for 
object  1  in  NRL-assigned  cell  L10  in  scenario  3,  for  which  the  X-Y  coordinate  was 
actually  in  cell  110,  almost  77  m  away. 

In  only  five  cases  did  the  different  grid  cell  designation  imply  a  different  status  of 
the  declaration  for  the  MTADS  test.  There  were  three  declarations  that  were  completely 
outside  the  scenario  boundaries:  1/A9/4,1  2/G2/2,  and  2/G5/2.  They  are  excluded  from 
further  analysis.  Had  they  been  incorporated,  all  would  have  been  false  alarms. 


1  Scenario/NRL  grid  cell  assignment/NRL  object  number  (within  the  grid  cell).  For  instance,  2/H3/3  is 
object  number  3  in  NRL-assigned  grid  cell  H3  in  scenario  2.  Referring  to  Table  1-4,  2/H3/3  is  actually 
in  grid  cell  13. 
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Table  1-4.  Grid  Cell  Assignment  to  Declarations:  Disagreements 
Between  NRL  and  IDA.  They  Are  Listed  in  Order  of 
IDA  Grid  Cell  Assignment. 


NRL 

IDA 

Scenario 

Grid 

Object 

Number 

Grid 

Distance  to 
NRL-Assigned 
Boundary 

Comment 

1 

Aerial 

Gunnery 

A9 

4 

79 

2  cm 

Outside  Scenario  Boundary 

C8 

2 

B8 

59  cm 

B11 

5 

BIO 

55  cm 

BIO 

i 

B11 

12  cm 

C11 

12 

B11 

1.0  m 

C12 

14 

B12 

60  cm 

E10 

6 

DIO 

2  cm 

E10 

7 

DIO 

70  cm 

Dll 

6 

D12 

36  cm 

D12 

12 

D13 

2  cm 

D12 

13 

D13 

1.3  m 

D13 

8 

D14 

3.0  m 

E13 

7 

E14 

14  cm 

F9 

4 

F8 

24  cm 

E13 

3 

FI  3 

30  cm 

2 

Artillery  and 
Mortar 

G2 

2 

F2 

10  cm 

Outside  Scenario  Boundary 

G5 

2 

F5 

Outside  Scenario  Boundary 

J2 

2 

12 

91  cm 

H3 

3 

13 

6  cm 

H3 

8 

13 

4.4  m 

H3 

9 

13 

2.1  m 

H3 

10 

13 

5.7  m 

L2 

ii 

K2 

L2 

12 

K2 

3.3  m 

L4 

2 

K4 

92  cm 

L4 

6 

K4 

3.4  m 

L4 

7 

L5 

1  cm 

LI 

6 

Ml 

5  cm 

N2 

i 

M2 

18.5  m 

Large  Distance  to  Boundary 

N2 

2 

M2 

16.9  m 

Large  Distance  to  Boundary 

N2 

3 

M2 

29.2  m 

Large  Distance  to  Boundary 

N2 

4 

M2 

9.6  m 

Large  Distance  to  Boundary 

3 

Submunitions 
and  Grenades 

H11 

4 

H12 

46  cm 

113 

5 

H13 

3.2  m 

L10 

1 

110 

76.8  m 

Changed  to  a  Surveyed  Ceil 

K11 

4 

J11 

4.4  m 

Changed  to  a  Surveyed  Cell 

LI  2 

6 

K12 

37  cm 

LI  3 

1 

LI  2 

32  cm 
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There  were  two  declarations  (3/L10/1  and  3/K11/4)  in  scenario  3  that  were 
changed  from  NRL-assigned  grid  cells  that  would  have  corresponded  to  unsurveyed 
areas,  to  grid  cells  that  were  counted  as  surveyed.  These  declarations  are  included  in  our 
analysis. 

The  remainder  of  the  miscategorization  documented  in  Table  1-4  merely  shifted 
declarations  from  one  surveyed  cell  to  another  surveyed  cell,  hence  they  are  included  in 
the  analysis. 

Table  1-5  shows  the  cells  that  were  not  surveyed,2  along  with  the  number  of 
declarations  in  the  cells.  Thus,  of  the  271  declarations  in  scenario  1,  5  were  excluded, 
leaving  266  for  further  analysis.  Of  the  283  in  scenario  2,  23  were  excluded,  leaving  260. 
In  scenario  3,  19  were  excluded  from  242,  leaving  223.  We  used  749  declarations  from 
all  three  scenarios  in  our  analysis. 

The  totals  of  the  targets  that  NRL  declared  as  ordnance  or  nonordnance  in  each 
scenario  are  given  in  Table  1-6.  For  ease  of  comparison  to  the  information  provided  by 
NRL  in  Reference  1-2,  NRL’s  numbers  are  also  given  in  this  table,  in  the  “Original  Cell 
Assignments”  column.  The  differences  are  wholly  a  consequence  of  the  grid  cell 
assignment  differences  given  in  Table  1-5.  From  this  point  forward,  only  the  numbers 
from  the  correct  cell  assignment  are  used. 

In  addition,  one  of  the  declarations  in  scenario  2  is  removed  because  it  refers  to  a 
“grouped  target.”  A  grouped  target  is  a  pair  of  closely  spaced,  or  “grouped,”  ordnance.  It 
is  further  explained  in  the  discussion  of  the  baseline  items  in  the  next  chapter. 

Two  types  of  MTADS  sensors  were  used,  pulsed  EMI  and  magnetic  field 
measurement  (MAG).  Most  of  the  time  both  sensors  would  detect  the  supposed  ordnance, 
a  common  detection  (COM);  however,  at  other  times,  only  one  sensor  made  the 
detection.  The  breakdown  by  sensor  for  scenarios  1,  2,  3,  and  all  scenarios  is  given  in 
Figures  1-3  to  1-6,  respectively. 


2  Reference  1-2  specifies  the  cells  that  were  not  surveyed. 
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Table  1-5.  Declarations  in  Grid  Cells  That  Were  Not  Surveyed 


Scenario 

Grid  Cell 

NRL  Number  of 
Declarations  in  the 
Cell 

IDA  Number  of 
Declarations  in 
the  Cell 

Comments 

1 

A13 

2 

2 

A14 

2 

2 

2  cm  East  of  cell 
A9 

0 

1 

Outside  Scenario 
Boundary 

Scenario  1  Total 

4 

5 

2 

F2 

0 

i 

Outside  Scenario 
Boundary 

F5 

0 

1 

Outside  Scenario 
Boundary 

G1 

5 

5 

HI 

5 

5 

M6 

7 

7 

M7 

3 

3 

N7 

1 

1 

Scenario  2  Total 

21 

23 

3 

K11 

4 

3 

One  was  actually  in 
J11 

L10 

2 

1 

One  was  actually  in 
110 

LI  2 

5 

6 

LI  3 

3 

2 

5 

5 

N13 

2 

2 

Scenario  3  Total 

21 

19 

Table  1-6.  Target  Declarations  as  Ordnance  or  Nonordnance 


Scenario 

Declared 
Target  Type 

Corrected  Cell  Assignments 

Original  Cell  Assignments 

Number 

Surveyed 

Number 

Unsurveyed 

Number 

Surveyed 

Number 

Unsurveyed 

1 

Aerial  Gunnery 

Ordnance 

185 

4 

186 

3 

Non-ordnance 

81 

1 

81 

1 

2 

Artillery  and 
Mortar 

Ordnance 

217(216)* 

13 

218 

12 

Non-ordnance 

43 

10 

44 

9 

3 

Submunitions  s 
and  Grenades 

Ordnance 

215 

19 

213 

21 

Non-ordnance 

7 

0 

7 

0 

*  The  number  in  parentheses  excludes  a  declaration  on  a  grouped  target.  See  next  chapter  for  further  explanation. 
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Figure  1-3.  Declarations  by  Sensor  Type  for  Scenario  1 — Aerial  Gunnery 


Figure  1-4.  Declarations  by  Sensor  Type  for  Scenario  2 — Artillery  and  Mortar 


Figure  1-5.  Declarations  by  Sensor  Type  for  Scenario  3 — 
Submunitions  and  Grenades 


Figure  1-6.  Declarations  by  Sensor  Type  for  All  Scenarios 
C.  TEST  DATA 

All  the  test  data  were  provided  by  NRL  in  the  form  of  three  spreadsheets.  These 
data,  as  specified  in  Ref.  1-2  and  described  in  this  chapter,  are  used  in  our  analysis. 
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II.  ANALYSIS  OF  NRL  DECLARATIONS 


This  chapter  presents  a  detailed  analysis  of  the  MTADS  detections.  The  analysis  is 
predicated  on  NRL’s  declaration  of  each  detection  as  ordnance  or  nonordnance,  as  out¬ 
lined  in  Chapter  I.  The  analysis  is  presented  in  the  following  order: 

•  Description  of  the  known  baseline  items 

•  Performance,  including  probability  of  detection  and  false-alarm  rates 

•  Sensor  performance  characteristics,  including  location  accuracy  and  correla¬ 
tion  among  sensors  (EMI  and  MAG). 

A.  THE  BASELINE 

Baseline  items  are  those  items  that  were  purposefully  placed  in  the  ground  at  the 
North  Test  Site  at  JPG.  This  baseline  includes  both  ordnance  and  nonordnance  items.  In 
the  following  analysis,  the  MTADS  performance  is  evaluated  against  the  baseline  items. 
There  is  no  guarantee  that  items  other  than  the  baseline  items  are  not  in  the  ground; 
indeed,  it  is  expected  that  there  are  such  items.  With  this  in  mind,  this  analysis,  strictly 
speaking,  is  an  exercise  in  setting  performance  bounds.  Specifically,  performance  is  evalu¬ 
ated  only  with  respect  to  the  known  baseline  items,  given  in  Table  II- 1.  The  density  of 
UXO  is  conveniently  expressed  in  terms  of  number  per  hectare.  (A  hectare  is  10,000  m2, 
or  about  2.5  acres.) 


Table  11-1.  Number  of  Baseline  Items  in  the  Surveyed  Areas 


Scenario 

Number  of  UXO 

Density  of  UXO  (per  hectare) 

Ordnance 

Nonordnance 

Total 

Ordnance 

Nonordnance 

Total 

1 

47 

78 

125 

14.1 

23.3 

37.4 

2 

75  (73)* 

50 

125(123)* 

19.1  (18.5)* 

12.7 

32(31)* 

3 

86 

38 

124 

28.9 

12.8 

41.7 

All 

208  (206)* 

166 

374  (372)* 

20.3  (20.1)* 

16.2 

36.5  (36.3)* 

*  Two  ordnance  were  within  R '  of  each  other  and  were  excluded  for  that  reason. 

cat 


One  pair  of  ordnance  in  scenario  2  was  within  the  critical  radius,  Rcrit,  of  1  m.  is 
the  nominal  resolution  of  the  sensors  and  can  be  thought  of  as  defining  a  detection 


“opportunity.”  That  is,  each  sensor  has  the  opportunity  to  detect  an  ordnance  item  for 
every  subsequent  area  of  1  m2  that  is  surveyed. 

There  is  some  ambiguity  attached  to  detection  of  ordnance  closer  than  ROT.(  to 
another  ordnance  item.  Should  a  detection  of  one  be  considered  a  detection  of  the  other, 
that  is,  two  detections?  Or,  should  it  be  considered  a  single  detection  of  a  pair,  that  is,  one 
detection?  Rather  than  attempting  a  definitive  resolution,  we  avoided  this  issue  by  exclud¬ 
ing  all  grouped  ordnance  (see  Table  II-2)  from  the  analysis.  This  is  consistent  with  the 
recent  analysis  of  the  MTADS  performance  at  Twentynine  Palms,  California  (Ref.  II- 1). 


Table  11-2.  Grouped  Ordnance  (from  scenario  2) 


Type  Ordnance 
#1 

Type  Ordnance 
#2 

Surface  Distance 
Between 
Ordnance  (m) 

Depth  of 
Ordnance  #1 
(m) 

Depth  of 
Ordnance  #2 
(m) 

81 -mm  mortar 
(illumination 
case) 

8-in.  projectile 

0.77 

0.23 

0.61 

B.  GENERAL  PERFORMANCE  RESULTS 

We  compared  the  MTADS  data  sets  from  the  three  scenarios  to  the  items  in  the 
baseline.  To  match  a  baseline  item  (ordnance  or  nonordnance)  to  a  declaration,  the 
distance  in  the  plane  of  the  Earth’s  surface,  the  two-dimensional  radial  distance,  r,  derived 
from  the  relative  northing  and  easting  offsets,  must  be  within  the  critical  radius,  RcriI,  of  the 
center  of  the  emplaced  item.  If  the  declaration  is  classified  by  NRL  as  ordnance,  Rcrit  ^  r, 
and  the  item  in  the  baseline  is  ordnance,  it  is  a  detection.  If  r  >  Rm[  or  the  baseline  item  is 
nonordnance,  the  declaration  is  a  false  alarm.  (See  Figure  II- 1.) 

1.  Probabilities  of  Detection  and  FARs 

Detection  probabilities  (PD s)  and  false-alarm  rates  (FARs)  were  determined  for  the 
three  scenarios,  individually  and  in  aggregate.  Table  El-3  gives  the  PD s.  Table  II-4  gives 
the  FARs.  An  Rcrit  of  1.0  m  is  used  throughout.  The  results  here  combine  the  sensors  such 
that  if  any  detection  by  any  sensor  (MAG  or  EMI)  is  within  ROT[,  it  is  considered  a 
detection,  hereafter  referred  to  as  an  “overall”  detection. 

More  stringent  requirements  are  examined  in  detail  later.  For  the  moment,  the 
effect  of  greater  stringency  may  be  usefully  gauged  by  requiring  that  both  sensors  achieve 
a  satisfactory  detection.  This  PD  is  given  in  the  last  column  of  Table  II-3. 
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Figure  11-1.  Detection  and  False-Alarm  Dependence  on  Rcrir  Detection  of  the  ordnance  item 
is  dependent  on  the  relative  location  of  the  declaration,  marked  with  a  four-pointed  star, 
and  the  center  of  volume  of  the  item.  If  the  item  is  ordnance  and  the  declaration  is 
within  the  critical  radius,  the  ordnance  is  detected.  If  the  declaration  is 
nonordnance,  not  in  the  baseline,  or  outside  Reiil,  it  is  a  false-alarm. 


Table  11-3.  MTADS  Overall  Detection  Probability  at  Jefferson  Proving  Ground 


Scenario 

Number  of 
Ordnance 

Number  Detected  By 

Overall 

Number 

Detected 

Overall 

PD[%] 

Restrictive 
PD  (both 
sensors 
required) 

EMI 

Only 

MAG 

Only 

Both  MAG 
and  EMI 

1 

47 

15 

0 

30 

45 

95.7 

63.8 

2 

73 

5 

1 

64 

70 

95.9 

87.7 

3 

86 

54 

0 

26 

80 

93.0 

30.2 

All 

206 

74 

1 

120 

195 

94.7 

58.3 

We  also  studied  more  restrictive  subsets  of  declarations:  (1)  declarations  that  had 
at  least  one  magnetometer  location  datum  (a  MAG  declaration),  and  (2)  declarations  that 
had  at  least  one  EMI  location  datum  (an  EMI  declaration).  These  subsets  focus  on  the 
relative  contribution  of  MAG  versus  EMI. 


II-3 


Table  11-4.  MTADS  Overall  False-Alarm  Rates  at  Jefferson  Proving  Ground 


Scenario 

Declaration 
Not  in 
Baseline 

Nonordnance  Baseline  Items 
Declared  Ordnance — Detection 
by: 

Total 
Number 
of  False 
Alarms 

FAR 

(per 

hectare) 

P, 

EMI 

only 

MAG 

only 

Both 
MAG 
and  EMI 

1 

81 

0 

3 

56 

140 

41.9 

1.32x1 0‘2 

2 

100 

9 

0 

37 

146 

37.6 

1.18  x  10‘2 

3 

110 

8 

3 

14 

135 

45.4 

1.43  x10‘2 

All 

291 

17 

6 

107 

421 

41.1 

1.29x1  O’2 

We  also  studied  detections  and  false  alarms  of  declarations  that  were  either 
exclusively  MAG  or  exclusively  EMI.  It  turns  out  that  the  main  issues  as  far  as  PD  go  are 
already  apparent  in  the  (nonexclusive)  MAG  and  (nonexclusive)  EMI  presentations  given 
next.  On  the  other  hand,  the  mutually  exclusive  MAG-EMI  data  well  illustrate  trends  in 
FAR.  Therefore,  it  is  presented  in  the  corresponding  section,  section  3  below. 

a.  EMI  Declarations 

EMI  declarations  include  those  with  only  EMI  ordnance  coordinates  and  those 
with  both  EMI  and  MAG  ordnance  coordinates.  Tables  II-5  and  II-6  summarize  this  cut  of 
the  data,  relevant  to  PD  and  FAR,  respectively.  Most  prominent  is  the  PD  for  scenarios  1 
and  3  (colored  cells  in  Table  H-5).  It  is  the  same  as  the  PD  for  both  sensors  combined 
(compare  to  Table  II-3).  Even  more  impressively,  the  FAR  decreased  significantly,  as 
shown  by  comparing  the  colored  cells  in  Table  II-6  to  the  corresponding  data  in  Table 
II-4.  In  these  scenarios  it  would  have  been  better  not  to  have  used  the  magnetometer  at  all. 


Table  11-5.  MTADS  Detection  Probabilities  for  the  EMI  Sensor 


Scenario 

Number  of 
Ordnance 

Number  Detected  By 

Overall 

Number 

Detected 

Overall 

PD[%) 

Restrictive 
PD  (both 
sensors 
required) 

EMI 

Only 

MAG 

Only 

Both 

MAG  and 
EMI 

1 

47 

15 

0 

30 

45 

1  95.7 

63.8 

2 

73 

5 

0 

64 

69 

87.7 

3 

86 

54 

0 

26 

80 

93.0 

30.2 

206 

74 

120 

194 

58.3 
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Table  11-6.  MTADS  FARs  for  the  EMI  Sensor 


Scenario 

Declaration 
Not  in 
Baseline 

Nonordnance  Baseline  Items 
Declared  Ordnance— Detection 
by: 

Total 

Number  of 
False 
Alarms 

FAR 

[per 

hectare] 

P, 

EMI  only 

MAG  only 

Both 

MAG  and 
EMI 

1 

22 

0 

3 

56 

81 

ME i 

7.6  x  10* 

2 

34 

9 

0 

37 

80 

20.3 

6.4x10* 

3 

68 

8 

2 

14 

92 

1  31.0 

9.7x10-* 

All 

124 

17 

5 

107 

253 

24.7 

IQ-3 

b.  MAG  Declarations 


MAG  declarations  include  declarations  with  only  MAG  ordnance  coordinates  and 
those  with  both  EMI  and  MAG  ordnance  coordinates.  Tables  II-7  and  II-8  summarize  this 
cut  of  the  data,  relevant  to  PD  and  FAR,  respectively. 

c.  False  Alarms  for  Declarations  Not  Common  to  Magnetometer  and  EMI 

False  alarms  are  examined  several  ways  in  this  document.  In  this  section,  an 
analysis  is  done  in  terms  of  declarations  made  exclusively  by  one  or  the  other  sensor. 


Table  11-7.  MTADS  Detection  Probabilities  for  the  MAG  Sensor 


Scenario 

Number  of 
Ordnance 

Number  Detected  By 

Overall 

Number 

Detected 

Overall 

Po  [%] 

Restrictive 
PD  (both 
sensors 
required) 

EMI 

Only 

MAG 

Only 

Both  MAG 
and  EMI 

1 

47 

2 

0 

30 

32 

68.1 

63.8 

2 

73 

1 

1 

64 

66 

B1SI 

87.7 

3 

86 

1 

0 

26 

27 

MEM 

30.2 

All 

206 

4 

1 

120 

125 

60.7 

58.3 
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Table  11-8.  MTADS  FARs  for  the  MAG  Sensor 


Scenario 

Declaration 
Not  in 
Baseline 

Nonordnance  Baseline  Items 
Declared  Ordnance  -  Detection  by: 

Total 
Number 
of  False 
Alarms 

FAR 

[per 

hectare] 

EMI 

only 

MAG 

only 

Both 
MAG 
and  EMI 

1 

71 

0 

3 

56 

130 

38.9 

1.22x1  O'2 

2 

89 

2 

0 

37 

128 

32.5 

1.02  x  10‘2 

3 

70 

0 

3 

14 

77 

25.9 

8.1  x  10^ 

All 

230 

2 

6 

107 

345 

33.6 

1.06x1  O’2 

Subtle  issues  attend  the  analysis  of  false  alarms.  These  are  mostly  a  consequence 
of  reasonably  defining  an  “opportunity,”  the  denominator  in  P/a. 


#  false  alarms 

#  opportunities 


(IM) 


As  done  in  Reference  II- 1,  a  resolution  may  be  reached  by  convention,  in  this  case 
defining  a  nominal  opportunity  in  terms  of  a  circular  area  around  a  declaration  with  radius 
Rcrii.  The  total  number  of  opportunities  is  then 

A  ypn 

# opportunities  = - .  (H-2) 

^crit 


For  a  high  density  of  false  alarms,  the  overlap  of  the  areas  associated  with  each  false  alarm 
can  be  significant.  Correcting  for  the  overlap  yields  a  definition  of  Pfa  in  terms  of  “the  area 
of  the  false- alarm  declarations,”  A^,  and  the  total  surveyed  area: 

r„=  .  A*  (n-3) 

Areasuneyed 


In  the  present  case  of  low  Pfa,  the  overlaps  are  negligible,  so 

Afa  =  #  false  alarms  x  nRlHt  ,  (II-4) 


which  implies 

Pfi  =  FAR  x  7tRlHt 


where 


.  _  #  false  alarms 

FAR  =  — - 


Area 


'surveyed 


(H-5) 


(II-6) 
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This  limiting  case  is  used  here,  but  strictly  speaking,  the  Pfa  so  calculated  should  be 
regarded  as  a  surrogate  for  the  true  Pfa. 

The  false  alarms  due  exclusively  to  one  sensor  are  given  in  Tables  II-9  and  II- 10, 
for  EMI-exclusive  declarations  and  MAG-exclusive  declarations,  respectively.  The  EMI 
does  better  on  false  alarms  except  for  scenario  3,  where  the  FARs  are  comparable  (16.2 
per  hectare  for  EMI  and  14.5  per  hectare  for  MAG).  However,  the  difference  in  their 
contribution  to  PD  is  dramatic.  The  contribution  is  called  A PD  here  because  it  is  in  addition 
to  the  detections  that  EMI  and  MAG  had  in  common. 


A PD  for  each  sensor  is  defined  as 


APd  for  EMIfMAG)  = 


#  of  EMIQMAG)  declarations  that  result  in  a  declaration 
#  of  baseline  ordnance 


Thus,  using  EMI  as  an  example, 

AP0(EMI)  from  Table  II-9  +  APD(MAG)  from  Table  II-7  =  Overall  PD  from  Table  H-3. 
Or,  numerically,  for  Scenario  1, 

—  from  Table  II  -  9  +  —  from  Table  II  -  7  =  —  from  Table  II  -  3  . 

47  47  47 

Similarly,  A FAR  for  each  sensor  is  defined  as 

.  „ .  „ .  ,  A  „  #  EMI(M  AG)  -  only  declarations  that  result  in  a  false  alarm 

AFAR  for  EMI(MAG)  s - - - - - - - - -  , 

area  surveyed 

so  that,  using  EMI  as  an  example, 

AFAR(EMI)  +  FA/?(EMI)  =  overall  FAR  . 

Referring  to  Scenario  1  and  Tables  II-9,  II-8,  and  El-4,  as  an  example,  gives 

3.0  +  38.9  =  41.9  , 

as  it  should. 

A PD  hovers  around  1  percent  for  MAG,  but  reaches  over  60  percent  for  EMI. 
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Table  11-9.  Detections  and  False  Alarms  for  EMI-Exclusive  Declarations 


Scenario 

Number 

Detected 

A  PD 
[%] 

Not  In 
Baseline 

Not 

Ordnance 

Total 

False 

Alarms 

A  FAR 
[per 

hectare] 

A  P„ 

1 

13 

28 

10 

0 

10 

3.0 

9.4x10^ 

2 

4 

5.5 

11 

7 

18 

4.6 

1.44  xIO"3 

3 

53 

62 

40 

8 

48 

16.2 

5.1  x  10"3 

All 

70 

34 

61 

15 

76 

7.4 

2.3  x  1 0'3 

Table  11-10.  Detections  and  False  Alarms  for  MAG-Exclusive  Declarations 


Scenario 

Number 

Detected 

APd 

[%] 

Not  In 
Baseline 

Not 

Ordnance 

Total 

False 

Alarms 

A  FAR 
[per 

hectare] 

A  P> 

1 

0 

0 

59 

0 

59 

17.6 

5.5x1  O’3 

2 

1 

1.4 

66 

0 

66 

16.8 

5.3x1  O'3 

3 

0 

0 

42 

1 

43 

14.5 

4.5x10-* 

All 

1 

0.5 

167 

1 

168 

16.4 

lO'3 

d.  Introduction  to  the  Baseline  Partitions  and  the  Declaration  Partitions 

There  are  two  complementary  ways  to  look  at  detection  effectiveness:  as  a 
partition  of  the  baseline  items  or  as  a  partition  of  the  declarations.  Partitioning  the  baseline 
ordnance  items  into  detected  and  undetected  classes  gives  the  PD  measure  of  effectiveness 
(MOE).  Partitioning  the  number  of  declarations  of  ordnance  into  those  that  are  correct  and 
those  that  are  erroneous  yields  the  Pfa  MOE.  These  partitions  are  a  systematic  display  of 
the  data  as  (inverted)  classification  trees,  sometimes  called  dendrograms  or  dendritics,  but 
referred  to  here  simply  as  trees. 

Different  kinds  of  analyses  of  this  data  could  also  be  done.  One  could  view  the 
MOE  as  a  chain  of  conditional  probabilities,  and  examine  the  chain  to  explore  where  the 
problem  of  producing  better  MOEs  lies.  One  could  also  ask  whether  other  classification 
trees  might  be  more  informative  about  the  MOE.  CART  (Classification  and  Regression 
Trees)  is  a  well-developed  statistical  method  for  building  and  using  such  trees.  Neither  of 
these  methods  is  further  pursued  here.  The  conditional  probabilities  are  evident  in  the  trees 
themselves,  and  are  conventionally  taken  as  the  probability  to  detect  (given  a  target)  and 
the  probability  to  classify  (given  a  detection).  CART  and  other  classification  techniques 
may  prove  informative  but  are  beyond  the  scope  of  the  present  study. 
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2.  Scenario  1 


The  analysis  of  scenario  1,  aerial  gunnery,  is  presented  in  two  parts.  The  first 
section  presents  partitions  of  the  important  data  sets  and  gives  the  essential  results  of  the 
test.  The  last  section  explores  false-alarm  mitigation  using  linear  statistical  discrimination. 

a.  Partitions  of  the  Baseline  and  the  Declaration  Set 

Figure  31-2  shows  the  partition  of  the  125  baseline  items  in  scenario  1.  The 
baseline  is  the  relevant  set  for  determining  PD.  Specifically,  the  47  ordnance  in  the 
baseline  constituted  the  detection  opportunities.  There  were  46  “detections”1  in  one  of  the 
three  categories  of  declarations:  EMI,  MAG,  or  both  (COM).  There  was  no  declaration  by 
MAG  alone  that  corresponded  to  an  actual  detection.  By  contrast,  EMI  made  a  third 
(15/46)  of  the  detections  by  itself. 

In  a  similar  manner.  Figures  II-4  and  II-6  give  the  irrelevant  information  by 

sensor. 


Figure  H-3  shows  the  partition  of  the  266  declarations  relevant  for  determination 
of  the  FAR.  Specifically,  the  185  items  declared  as  likely  ordnance  each  can  end  up  being 
a  false  alarm.  Of  the  104  “detections”  in  the  baseline,  15  were  by  EMI  alone,  3  solely  by 
MAG,  and  86  by  both  sensors.  The  number  of  false  alarms  was  the  sum  of  those 
“detections”  of  nonordnance  and  the  “detection”  of  items  not  in  the  baseline.  That  is,  there 
were  140  false  alarms,  the  components  of  which  are  shown  colored  in  Figure  H-3.  EMI  by 
itself  did  not  contribute  any  false  alarms  arising  from  baseline  nonordnance.  There  were, 
however,  10  nonbaseline  items  detected  only  by  EMI  that  were  false  alarms  (see  Table 
II- 11). 

Figures  II-5  and  II-7  give  the  false  alarm  (FA)-relevant  information  by  sensor. 


1  Quotes  distinguish  “detection”  of  some  object  in  the  baseline,  be  it  ordnance  or  nonordnance,  as 
opposed  to  detection  without  quotes,  which  refers  (in  most  of  this  document  and  in  UXO  work 
generally)  to  detection  of  only  baseline  ordnance  and  does  not  include  “detection”  of  nonordnance. 


H-9 


125 

Baseline  it< 


47 

Ordnance 


76 

Nonordnance 


r — 

0 

0 

Didarod 

Ds  dared 

Declared 

Nojford  nance 

Ordnance 

Nonordnanoe 

Fit  ure  11-2.  Partition  of  the  Baseline  for  the  PD  MOE:  All  Sensor  Declarations  for  the  1st 
Scenario,  Aerial  Gunnery.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  PD.  The  EMI,  MAG,  and  COM  boxes  indicate  the  number  of 
declarations  by  EMI  alone,  MAG  alone,  and  by  both  MAG  and  EMI 
(that  is,  common  declarations). 
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Figure  11-3.  Partition  of  the  Declarations  for  the  FAR  MOE:  All  Sensor  Declarations  for  the 
1st  Scenario,  Aerial  Gunnery.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  the  FAR.  The  P„  is  calculated  assuming  that  an  opportunity 
occurs  for  every  circular  area  with  radius  of  R^,.  A 1  -m  RcHt  is  used.  The  EMI,  MAG,  and 
COM  boxes  indicate  the  number  of  detections  achieved  by  EMI  alone,  MAG  alone, 
and  by  both  MAG  and  EMI  (that  is,  common  detections). 
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Figure  11-4.  Partition  of  the  Baseline  for  the  PD  MOE:  EMI  Declarations  for  the  1st  Scenario, 
Aerial  Gunnery.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the  declarations  that 
contribute  to.  The  EMI,  MAG,  and  COM  boxes  indicate  the  number  of  declarations  by  EMI 
alone,  MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  declarations). 


Figure  11-5.  Partition  of  the  Declarations  for  the  FAR  MOE:  EMI  Declarations  for  the  1st 
Scenario,  Aerial  Gunnery.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  the  FAR.  The  P,a  is  calculated  assuming  that  an 
opportunity  occurs  for  every  circular  area  with  radius  of  R^,.  A  1-m  RM  is  used. 

The  EMI,  MAG,  and  COM  boxes  indicate  the  number  of  detections  achieved  by 
EMI  alone,  MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  detections). 
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Figure  11-6.  Partition  of  the  Baseline  for  the  P0  MOE:  MAG  Declarations  for  the  1st  Scenario, 
Aerial  Gunnery.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the  declarations  that 
contribute  to  PD.  The  EMI,  MAG,  and  COM  boxes  indicate  the  number  of  declarations  by  EMI 
alone,  MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  declarations). 


Figure  11-7.  Partition  of  the  Declarations  for  the  FAR  MOE:  MAG  Declarations  for  the  1st 
Scenario,  Aerial  Gunnery.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  the  FAR.  The  P,a  is  calculated  assuming  that  an 
opportunity  occurs  for  every  circular  area  with  radius  of  RM.  A  1-m  Rcrit 
is  used.  The  EMI,  MAG,  and  COM  boxes  indicate  the  number  of 
detections  achieved  by  EMI  alone,  MAG  alone,  and  by  both 
MAG  and  EMI  (that  is,  common  detections). 

Table  II- 11  shows  how  the  nonbaseline  declarations  break  down.  All  information 
in  Table  II- 11  can  be  derived  from  the  trees.  For  example,  the  12  common  declarations  for 
nonbaseline  items  that  appears  in  the  ordnance  declarations  in  Table  11-11  can  be  derived 
from  the  tree  information,  as  follows. 

1.  Add  the  22  EMI  ordnance  declarations  not  in  the  baseline  (Figure  II-5)  to  the 
71  MAG  ordnance  declarations  not  in  the  baseline  (Figure  II-7). 

2.  Take  the  total,  93,  and  subtract  the  81  ordnance  declarations  by  all  sensors 
(Figure  II-3). 

3.  The  difference,  12,  appears  as  the  number  of  common  declarations  in 
Table  II- 11. 
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The  other  data  in  Table  H- 11  can  be  similarly  deduced:  they  are  displayed  here  for 


clarity. 


Table  11-11.  Sensor  Relation  to  “Detected”  Object  Type  for  Scenario  1 


Actual  Object 
Type 

Declaration  Sensor 

Total 

MAG 

Common 

EMI 

Declared  as 
Ordnance 

Non-baseline 

59 

12 

10 

81 

Non-ordnance 

0 

59 

0 

59 

Ordnance 

0 

32 

13 

45 

Total  Declared 
Ordnance 

59 

103 

23 

185 

Declared  as  Non- 
Ordnance 

Non-baseline 

47 

28 

0 

75 

Non-ordnance 

0 

5 

0 

5 

Ordnance 

0 

1 

0 

1 

Total  Declared 
Ordnance 

47 

34 

0 

81 

b.  False-Alarm  Mitigation:  Correlation  and  Discriminant  Analysis 

In  this  section,  the  issue  of  discriminating  FAs  from  detections  for  the  purpose  of 
reducing  the  FAR  is  explored. 

The  method  of  analysis  is  straightforward  classical  linear  discrimination,  as 
described  in  Reference  II-2  (Chapter  3)  or  Reference  2  (Chapter  11).  In  particular,  the 
adoption  of  Fisher’s  linear  discriminant  (Ref.  II-2,  p.  93)  enables  a  simple  maximization 
of  the  ratio  of  the  between-group  variance  to  the  within-group  variance,  a  reasonable  goal 
even  in  the  absence  of  multivariate  normality.  On  the  other  hand,  if  multivariate  normality 
obtains,2  Fisher’s  rule  follows  from  classical  decision  theory  (Ref.  II- 2,  Section  11.3). 
Here,  the  groups  are  MTADS’  ordnance  declarations  that  correspond  to  ordnance,  called 
“Detections,”  and  such  declarations  that  do  not,  called  “False  Alarms.” 

We  conducted  discriminant  analyses  on  four  batches  of  data.  The  first  three 
correspond  to  the  individual  scenarios;  the  fourth  batch  is  an  aggregation  of  all  three 
scenarios.  It  was  analyzed  first  with  the  hope  of  identifying  a  discriminant  that  is  robust 
across  scenarios.  None  was  found.  Failing  that,  this  linear  discriminant  analysis  was  used 
to  see  if  the  scenario  could  be  identified  solely  by  using  the  data.  This  also  was  not 


2  For  the  linear  discriminant,  it  is  also  necessary  to  assume  equal  covariance  matrices. 
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possible  at  this  level  of  examination.  More  sophisticated  techniques  could  be  brought  to 
bear,  but  this  is  not  within  the  scope  of  this  document. 

The  variables  (in  statistical  parlance  “factors”)  that  we  considered  are  those  to 
which  the  MTADS  operator  might  conceivably  have  access.  For  the  MTADS 
magnetometer  these  are  fitted  depth  (ZMAG),  fitted  radius  (RMAG),  magnetic  moment 
(MAGMOM),  fitted  inclination  (INCLIN),  fitted  azimuthal  angle  (AZIM),  and  the  GoF 
(GOF_MAG).  For  the  MTADS  EMI  detection  they  are  fitted  depth  (ZEM),  ferrous  radial 
size  (RFE_EM),  nonferrous  radial  size  (RAL_EM),  and  GoF  (GOF_EM).  The  aim  here  is 
to  examine  variables  that  could  be  used  to  mitigate  FAs.  When  a  case  does  not  have  a 
required  variable,  it  is  omitted  from  the  analysis  altogether.  The  inclusion  of  variables, 
such  as  the  GoF,  which  are  currently  the  result  of  a  rather  elaborate  fitting  process,  is 
justified  by  anticipating  that  a  more  expeditious  method  of  producing  them,  or  close 
surrogates,  could  be  achieved.  Whether  this  is  in  fact  possible  or  practical  is  an  issue  for 
further  study. 

An  initial  survey  of  the  data  aggregated  from  all  scenarios  gives  the  variable 
correlations  shown  in  Table  II- 12.  There  were  298  declarations  with  all  data  available. 


Table  11-12.  Correlation  Matrix  for  All  Scenarios. 

298  declarations  had  all  data  available. 


ZMAG 

ZEM 

RMAG 

RFE_EM 

RAL_EM 

MAGMOM 

INCLIN 

AZIM 

GOF_MAG 

ZMAG 

1 

1 

1 

RFE_EM 

0.687 

1 

RAL.EM 

0.682 

0.807 

0.774 

0.989 

1 

MAGMOM 

0.529 

0.633 

0.667 

1 

INCLIN 

0.176 

0.04 

0.038 

-0.093 

1 

AZIM 

-0.061 

-0.056 

0.022 

0.039 

0.036 

-0.053 

-0.16 

1 

GOF_MAG 

0.136 

0.187 

0.362 

0.302 

0.266 

0.194 

-0.177 

0.051 

1 

GOF_EM 

0.192 

0.207 

0.371 

0.451 

0.362 

0.198 

0.04 

0.073 

0.347 

The  largest  correlations  are  for  (RFE_EM,  RALJEM)  with  correlation  0.989  and 
(RMAG,  MAGMOM)  with  correlation  0.834.  Scatterplots  of  these  pairs  (Figures  II-  8  and 
II-9,  respectively)  show  that  the  variables  are  completely  dependent.  Thus,  only  one  of 
each  pair  is  used  in  the  following  analysis.  The  variable  of  each  pair  that  is  kept  has  a 
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“more  even”  distribution  of  hits.  From  the  histogram  in  Figure  13-9,  it  is  clear  that  RMAG 
is  preferred,  the  distribution  of  MAGMOM  being  much  more  peaked  and  skewed. 


Like  consideration  of  Figure  II-8  indicates  a  preference  for  RFE_EM,  though  it  is 
not  so  pronounced.  This  is  because  the  relation  of  RFE_EM  to  RAL_EM  is  nearly  linear. 
Were  the  relation  completely  linear,  either  variable  would  be  equally  acceptable. 


RFE_EM  RAL_EM 


Figure  11-8.  Correlation  of  EMI  Ferrous  and  Nonferrous  Radii. 

Combined  data  with  50-percent  ellipses  are  shown. 
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Figure  11-9.  Correlation  of  Fitted  Magnetometer  Radius  and  Magnetic  Moment. 

Combined  data  with  50-percent  ellipses  are  shown. 
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b.  Discriminant  Analysis 

In  Scenario  1,  the  MTADS  operators  decided  that  185  of  the  266  declarations  were 
ordnance.  These  ordnance  declarations  are  shown  in  scatterplots:  Figure  II- 10  for 
magnetometer  variables,  Figure  II- 11  for  EMI  variables,  and  Figure  11-12  for  the  cross  of 
magnetometer  and  EMI  variables.  Table  11-13  shows  correlation  of  the  variables  in 
scenario  1. 


Table  11-13.  Correlation  of  Scenario  1  Variables.  103  Declarations 
for  Which  all  Data  Was  Available 


ZMAG 

ZEM  1 

RMAG 

INCLIN 

AZIM 

■cm  an 

ZMAG 

1 

ZEM 

0.724 

1 

RMAG 

0.696 

0.522 

1 

0.712 

0.807 

0.73 

1 

INCLIN 

0.32 

0.272 

-0.082 

0.237 

1 

AZIM 

-0.166 

-0.14 

-0.124 

-0.076 

-0.046 

1 

igaami 

0.322 

0.162 

0.447 

0.271 

0.271 

-0.051 

1 

IcfaW 

0.223 

-0.009 

0.363 

0.282 

0.282 

0.014 

0.309 

i 

ZMAG  RMAG  INCLIN  AZ1M  QOF_MAG 
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Figure  11-10.  Scenario  1  Correlation  of  Magnetometer  Variables 

Blue  indicates  false  alarms.  Red  indicates  detections. 
50-percent  ellipses  are  shown. 
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Figure  11-11.  Scenario  1  Correlation  of  EMI  Variables. 

Blue  indicates  false  alarms.  Red  indicates  detections. 
50-percent  ellipses  are  shown. 


ZMAG  RMAG  INCLIN  AZ1M  GOF_MAG 


Figure  11-12.  Scenario  1  Cross-Correlation:  EMI  vs.  MAG. 

Blue  indicates  false  alarms.  Red  indicates  detections. 
50-percent  ellipses  are  shown 


Figures  11-13  and  11-14  show  the  histograms  of  the  detections  and  false  alarms  with 
the  scale  of  the  independent  variable  indicated.  The  bins  are  not  at  the  same  positions  or  of 
the  same  width  as  those  in  the  previous  figures,  so  the  histogram  bar  heights  will  not 
coincide,  in  general. 


(b)  (e) 
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Figure  11-13.  Histograms  of  Detections  and  False  Alarms  for  the  Magnetometer 
Variables  for  Scenario  1.  (a)  Depth,  (b)  Radial  size,  (c)  Inclination, 

(d)  Azimuthal  angle,  (e)  GoF. 
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Figure  11-14.  Histograms  of  Detections  and  False  Alarms  for  the  EMI  Variables 
for  Scenario  1.  (a)  Depth,  (b)  Ferrous  Size,  (c)  GoF. 


The  discriminant  analysis  proceeds  from  consideration  of  all  the  independent 
variables,  both  MAG  and  EMI,  and  uses  backward  elimination  to  select  the  statistically 
significant  variables.  An  F  test3  is  used,  with  the  criteria  for  variable  elimination  being  a 
tail  probability  of  0.15.  This  procedure  gives  only  RFE_EM  as  statistically  significant  for 
scenario  1. 


Thus,  the  best  one  can  do,  according  to  this  analysis,  is  to  mitigate  false  alarms  by 
setting  a  threshold  on  RFE_EM.  The  RFE_EM  histogram  is  given  in  Figure  II- 14(b).  It  is 
not  greatly  encouraging.  Of  the  81  false  alarms,  only  13  can  be  mitigated  by  using  a 
ferrous  size  threshold  of  0.025  m.  The  prescription  would  be  to  classify  all  objects  with 


3  An  F  test  uses  the  ratio  of  sample  variances  as  a  test  statistic.  Under  the  assumption  that  the  underlying 
variance  is  normally  distributed,  the  F  statistic  is  distributed  in  accordance  with  a  known  function,  the  F 
distribution.  In  this  case,  the  F  statistic  is  the  ratio  of  the  between-group  and  within-group  variances. 
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EMI  ferrous  size  less  than  0.025  m  as  nonordnance  in  this  scenario.  The  false  alarms  are 
thereby  reduced  from  81  to  68,  a  reduction  of  16  percent. 

The  RFE_EM  discriminant  was  derived  by  using  only  the  separation  of  the  means 
of  the  distributions  themselves  as  the  goal.  More  robust  discriminants  are  likely  available. 
For  instance,  inspect  the  GOF_EM  vs.  RFE_EM  plot  in  Figure  II- 1 1.  The  lower  left  region 
more  clearly  separates  the  false  alarms  from  the  detections,  as  indicated  by  the  50-percent 
ellipses.  But  by  just  using  the  separation  of  detection  and  false  alarm  means  as  a  criterion, 
the  EMI  GoF  does  not  make  the  cut.  It  is  easy  to  get  possible  FAR  reductions  of 
25  percent  or  more  by  using  statistically  insignificant  variables  in  this  way.  A  quadratic 
boundary  (instead  of  a  linear  one)  also  helps. 

We  avoided  using  variables  that  are  not  statistically  significant  in  this  work, 
principally  because  we  feel  it  amounts  to  overfitting  the  data.  The  point  here  is  that  a 
physical  model  or  more  exhaustive  testing  could  well  result  in  discriminants  (such  as 
GOF_EM)  that  could  achieve  greater  FA  mitigation.  The  procedure  adopted  here,  because 
it  is  conservative  in  its  assumptions,  likely  understates  the  FA  mitigation  that  could 
actually  be  achieved. 

Last,  it  should  be  noted  that  RFE_EM  is  only  available  for  declarations  that  have 
an  EMI  contribution  and  thus  is  not  a  usable  discriminant  for  MAG-only  declarations. 
There  were  140  FAs  for  Scenario  1.  RFE_EM  was  available  for  only  81  of  those.  There¬ 
fore,  the  true  reduction  of  false  alarms  would  seem  to  be  only 

—  (0. 16)=  9.3%  .  (II-7) 

140 

However,  as  noted  in  the  scenario  1  performance  results  section,  all  the  MAG-only 
declarations  were  false  alarms.  This  suggests  mitigating  false  alarms  in  this  type  of 
scenario  by  a  two-step  process. 

1.  Disregard  all  declarations  for  which  only  magnetometer  performance  is 
available, 

2.  For  the  remaining  declarations,  EMI  information  is  available.  Apply  the 
RFE_EM  discriminant  here,  reducing  FAs,  by  an  additional  16  percent. 

Thus,  a  better  estimate  of  FA  mitigation  may  be 

J*1(0.16)+  —  =  51.4%  .  (II-8) 

140 V  '  140 
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3.  Scenario  2 


We  present  our  analysis  of  scenario  2,  artillery  and  mortar,  in  two  parts.  First,  we 
partition  the  important  data  sets  to  give  the  essential  results  of  the  test.  Second,  we  explore 
false-alarm  mitigation  using  linear  statistical  discrimination. 

a.  Partitions  of  the  Baseline  Set  and  the  Declaration  Set 


Figure  11-15  shows  the  partition  of  the  123  scenario  2  baseline  items.  The  73 
ordnance  are  relevant  for  determining  PD.  Similarly,  Figures  11-17  and  11-19  give  the 
PD-  relevant  information  by  sensor. 

Figure  11-16  shows  the  partition  of  the  259  scenario  2  declarations.  The  216 
declared  as  ordnance  are  relevant  for  detecting  the  FAs. 


Figures  11-18  and  11-20  give  the  FA-relevant  information  by  sensor. 

Table  11-14  shows  the  explicit  relation  of  the  declaration  type  to  the  “detected” 


object  (nonbaseline  means  the  object  type  was  not  known). 


Figure  11-15.  Partition  of  the  Baseline  for  the  PD  MOE:  All  Sensor  Declarations  for  the  2nd 
Scenario,  Artillery  and  Mortar.  The  yellow  boxes  (with  shadow  and  bold  type) 
contain  the  declarations  that  contribute  to  PD.  The  EMI,  MAG,  and  COM  boxes 
indicate  the  number  of  declarations  by  EMI  alone,  MAG  alone, 
and  by  both  MAG  and  EMI  (that  is,  common  declarations). 
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Figure  11-16.  Partition  of  the  Declarations  for  the  FAR  MOE:  All  Sensor  Declarations  for  the 
2nd  Scenario,  Artillery  and  Mortar.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  the  FAR.  The  P,„  is  calculated  assuming  that  an  opportunity 
occurs  for  every  circular  area  with  radius  of  Rchr  A  1-m  is  used.  The  EMI,  MAG, 
and  COM  boxes  indicate  the  number  of  “detections"  achieved  by  EMI  alone, 

MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  detections). 


Figure  11-17.  Partition  of  the  Baseline  for  the  PD  MOE:  EMI  Declarations  for  the  2nd  Scenario, 
Artillery  and  Mortar.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  PD.  The  EMI,  MAG,  and  COM  boxes  indicate 
the  number  of  declarations  by  EMI  alone,  MAG  alone,  and  by  both 
MAG  and  EMI  (that  is,  common  declarations). 
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Figure  11-18.  Partition  of  the  Declarations  for  the  FAR  MOE:  EMI  Declarations  for  the  2nd 
Scenario,  Artillery  and  Mortar.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  the  FAR.  The  Pb  is  calculated  assuming  that  an  opportunity 
occurs  for  every  circular  area  with  radius  of  A  1-m  Rcri,  is  used.  The  EMI,  MAG, 
and  COM  boxes  indicate  the  number  of  “detections”  achieved  by  EMI  alone, 

MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  detections). 


Figure  11-19.  Partition  of  the  Baseline  for  the  PD  MOE:  MAG  Declarations  for  the  2nd 
Scenario,  Artillery  and  Mortar.  The  yellow  boxes  (with  shadow  and  bold  type)  contain 
the  declarations  that  contribute  to  P„.  The  EMI,  MAG,  and  COM  boxes  indicate  the 
number  of  declarations  by  EMI  alone,  MAG  alone,  and  by  both  MAG  and  EMI 
(that  is,  common  declarations). 
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Figure  11-20.  Partition  of  the  Declarations  for  the  FAR  MOE:  MAG  Declarations  for  the  2nd 
Scenario,  Artillery  and  Mortar.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  the  FAR.  The  P„  is  calculated  assuming  that  an  opportunity 
occurs  for  every  circular  area  with  radius  of  Rcrlr  A  1-m  Rcrjt  is  used.  The  EMI,  MAG, 
and  COM  boxes  indicate  the  number  of  “detections”  achieved  by  EMI  alone, 

MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  detections). 


Table  11-14.  Sensor  Relation  to  “Detected”  Object  Type 
for  Scenario  2  Ordnance  Declaration 


Declaration  Sensor 

Total 

Declared  as 
Ordnance 

"Detected" 
Object  Type 

MAG 

Common 

EMI 

Nonbaseline 

66 

23 

11 

100 

Nonordnance 

0 

39 

7 

46 

Ordnance 

1 

65 

4 

70 

Total  Declared 
Ordnance 

67 

127 

22 

216 

Declared  as 
Nonordnance 

Nonbaseline 

29 

12 

1 

42 

Nonordnance 

0 

0 

1 

0 

Ordnance 

0 

0 

0 

0 

Total  Declared 
Nonordnance 

29 

12 

2 

43 

b.  False-Alarm  Mitigation:  Correlation  and  Discriminant  Analysis 

As  for  scenario  1,  this  section  puts  forward  a  simple  discriminant  analysis  for  the 
MTADS  declarations  that  the  NRL  analysts  believed  were  ordnance.  The  aim  is  to 
identify  promising  means  of  reducing  the  FAR.  Again,  we  use  Fisher’s  linear  discriminant. 

As  before,  the  redundant  variables,  magnetic  moment  (correlated  with  radius)  and 
nonferrous  size  (correlated  with  ferrous  size),  are  eliminated  from  the  analysis.  Table  11-15 
shows  an  initial  survey  of  the  correlations.  Table  11-15  is  constructed  from  the  127 
observations  for  which  all  variables  were  available.  Figure  n-21  shows  scatterplots  of  the 
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MAG  variables;  Figure  11-22,  the  EMI  variables;  and  Figure  11-23,  the  EMI  versus  the 
magnetometer  variable.  Figures  n-24  and  n-25  give  histograms  of  the  variables  to  provide 
a  readable  scale.4 

Table  11-15.  Correlation  of  Scenario  2  Variables.  103  Declarations  for 
Which  all  Data  Were  Available 


ZMAG  FMAG  INXIN  AZ1M  GOF_MAG 


ZMAG  FMAG  IN3LIN  A21M  GOF_MAG 


Figure  11-21.  Scenario  2  Correlation  of  Magnetometer  Variables.  Blue  indicates 
false  alarms.  Red  indicates  detections.  50-percent  ellipses  are  shown. 


4  The  bins  are  not  exactly  the  same  as  for  the  scatterplots,  giving  some  height  variations  relative  to  the 
columns  shown  in  Figures  11-16  through  U-17. 
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ZEM  RFE_EM  GOF_EM 


ZEM  RFE_EM  GOF_EM 

Figure  11-22.  Scenario  2  Correlation  of  EMI  Variables.  Blue  =  False  Alarms. 
Red  =  Detections.  50-percent  ellipses  are  shown. 
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ZMAG  RMAG  INCLIN  AZIM  GOF_MAG 


Figure  11-23.  Scenario  2  Correlation  of  EMI  Variables  with  Magnetometer 
Variables.  Red  =  Detections.  Blue  =  False  Alarms. 

50-percent  ellipses  are  shown. 
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Figure  11-25.  Scenario  2  EMI  Histograms 

The  discriminant  analysis  produces  a  linear  combination  of  the  statistically 
significant  variables.  It  is  defined  as 

Factor  =  -  5.254  +  1.111  ZMAG  +  0.015  INCLIN  +  0.05 1  GOF_EM  .  (II-9) 

Figure  n-26  is  a  histogram  of  detections  and  false  alarms.  Of  the  62  false  alarms 
for  which  all  the  variables  were  available,  37  could  have  been  eliminated  by  adopting 
Factor  =  -0.50  as  a  threshold  and  regarding  all  declarations  below  this  as  nonordnance. 

This  scheme  could  reduce  this  set  of  FAs  by  60  percent  (37/62). 
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Figure  11-26.  Scenario  2  Discriminant  for  Common  Declarations. 

Factor  is  defined  in  Eq.  11-9. 
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Figure  11-27  provides  another  illustration  of  the  interplay  of  these  variables.  The 
declarations  of  ordnance  that  were  fitted  to  be  deeper,  with  more  negative  inclination,  and 
a  poorer  EMI  GoF  were  more  likely  to  be  false  alarms.  These  are  the  blue  squares  in 
Figure  11-27;  they  tend  to  be  in  the  octant  that  is  lower  and  toward  the  viewer.  The 
threshold  on  Factor  corresponds  to  a  plane  in  Figure  II- 27  (not  shown)  that  best  separates 
the  false  alarms  from  the  detections. 


This  reduction  is  only  applicable  for  those  FAs  for  which  both  EMI  (its  GoF)  and 
MAG  (inclination  and  fitted  depth)  information  is  available.  A  separate  discriminant 
analysis  follows  for  those  cases  in  which  only  EMI  or  only  MAG  data  are  obtained. 

For  MAG-only,  there  were  67  declarations,  66  of  which  were  false  alarms.  With 
only  one  detection  the  statistics  are  too  poor  to  find  a  good  discriminant  in  this  case. 

Thus,  the  best  estimate  of  achievable  FAR  reduction  is 

—  (62  x  60%  +  66  x  0%  + 18  x  0%)=  25.5%  .  (II- 10) 

146 
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Figure  11-27.  Dependence  of  Detections  and  False  Alarms  on  Variables  in  the  Discriminant 
for  Declarations  Common  to  the  Magnetometer  and  EMI  for  Scenario  2. 

Blue  Cubes  =  False  Alarms.  Red  Spheres  =  Detections. 

The  weights  derive  from  the  62  common,  66  MAG-only,  and  the  18  EMI-only 
false  alarms. 

3.  Scenario  3 

We  present  our  analysis  of  scenario  3,  submunitions  and  grenades,  in  two  parts. 
First,  we  partition  the  important  data  sets  to  give  the  essential  results  of  the  test.  Second, 
we  explore  false-alarm  mitigation  using  linear  statistical  discrimination. 

a.  Partitions  of  the  Baseline  Set  and  the  Declaration  Set 

Figure  n-28  shows  the  partition  of  the  124  scenario  3  baseline  items.  The  86 
ordnance  are  relevant  for  determining  PD.  Similarly,  Figures  11-30  and  11-32  give  the 
P0-relevant  information  by  sensor.  Figure  11-29  shows  the  partition  of  the  222  scenario  3 
declarations.  The  215  declared  ordnance  are  relevant  for  determining  the  FAs.  Similarly, 
Figures  11-31  and  11-33  give  the  FA-relevant  information  by  sensor. 
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Figure  11-28.  Partition  of  the  Baseline  for  the  PD  MOE:  All  Sensor  Declarations  for  the  3rd 
Scenario,  Submunitions  and  Grenades.  The  yellow  boxes  (with  shadow  and  bold  type) 
contain  the  declarations  that  contribute  to  PaThe  EMI,  MAG,  and  COM  boxes  indicate 
the  number  of  declarations  by  EMI  alone,  MAG  alone,  and  by  both  MAG  and  EMI 

(that  is,  common  declarations). 


Figure  11-29.  Partition  of  the  Declarations  for  the  FAR  MOE:  All  Sensor  Declarations  for  the 
3rd  Scenario,  Submunitions  and  Grenades.  The  yellow  boxes  (with  shadow  and  bold 
type)  contain  the  declarations  that  contribute  to  the  FAR.  The  Pfe  is  calculated 
assuming  that  an  opportunity  occurs  for  every  circular  area  with  radius  of  Rcrit. 

A  1-m  R^  is  used.  The  EMI,  MAG,  and  COM  boxes  indicate  the  number 
of  “detections”  achieved  by  EMI  alone,  MAG  alone,  and  by  both 
MAG  and  EMI  (that  is,  common  detections). 
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Figure  11-30.  Partition  of  the  Baseline  for  the  PD  MOE:  EMI  Declarations  for  the  3rd  Scenario, 
Submunitions  and  Grenades.  The  yellow  boxes  (with  shadow  and  bold  type)  contain  the 
declarations  that  contribute  to  Pg.  of  declarations  by  EMI  alone,  MAG  alone, 
and  by  both  MAG  and  EMI  (that  is,  common  declarations). 
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Figure  11-31 .  Partition  of  the  Declarations  for  the  FAR  MOE:  EMI  Declarations  for  the  3rd 
Scenario,  Submunitions  and  Grenades.  The  yellow  boxes  (with  shadow  and  bold  type)  contain 
the  declarations  that  contribute  to  the  FAR.  The  Plg  is  calculated  assuming  that  an  opportunity 
occurs  for  every  circular  area  with  radius  of  A  1-m  R^,  is  used.  The  EMI,  MAG,  and 
COM  boxes  indicate  the  number  of  “detections”  achieved  by  EMI  alone,  MAG  alone, 
and  by  both  MAG  and  EMI  (that  is,  common  detections). 


Figure  11-32.  Partition  of  the  Baseline  for  the  P0MOE:  MAG  Declarations  for  the  3rd 
Scenario,  Submunitions  and  Grenades.  The  yellow  boxes  (with  shadow  and  bold 
type)  contain  the  declarations  that  contribute  to  PD.  The  EMI,  MAG,  and  COM 
boxes  indicate  the  number  of  declarations  by  EMI  alone,  MAG  alone, 
and  by  both  MAG  and  EMI  (that  is,  common  declarations). 


Figure  11-33.  Partition  of  the  Declarations  for  the  FAR  MOE:  MAG  Declarations  for  the  3rd 
Scenario,  Submunitions  and  Grenades.  The  yellow  boxes  (with  shadow  and  bold  type) 
contain  the  declarations  that  contribute  to  the  FAR.  The  Ph  is  calculated  assuming  that 
an  opportunity  occurs  for  every  circular  area  with  radius  of  RM.  A  1-m  is  used. 

The  EMI,  MAG,  and  COM  boxes  indicate  the  number  of  detections  achieved  by 
EMI  alone,  MAG  alone,  and  by  both  MAG  and  EMI  (that  is,  common  detections). 
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Table  11-16  shows  the  relation  of  declaration  type  to  the  “detected”  object 
(nonbaseline  means  the  object,  if  there  was  one,  was  not  known). 


Table  11-16.  Sensor  Relation  to  “Detected”  Object  Type  for 
Scenario  3  Ordnance  Declarations' 


Declaration  Sensor 

"Detected" 
Object  Type 

MAG 

Common 

EMI 

Total 

</) 

<0  O 

Nonbaseline 

42 

40 

om 

Nonordnance 

1 

16 

8 

25 

Ordnance 

0 

27 

53 

80 

CO 

o  12 

Total 

Declared 

Ordnance 

43 

■ 

101 

215 

© 

Nonbaseline 

4 

2 

0 

6 

<8  B 

■g  C 

Nonordnance 

0 

1 

0 

1 

Ordnance 

0 

!  o 

0 

o  ! 

«  X 

Q  o 

Total 

Declared 

Ordnance 

4 

3 

0 

7 

a  Two  of  these  declarations,  3/K12/1  and  3/L9/3  had  only  EMI  and  MAG  data 
for  the  Easting  and  Northing  coordinates,  but  no  data  for  any  other  variables 
(RMAG,  RFE_EM,  etc.)  A  third  declaration,  3/HI 3/2,  had  only  EMI  variables 
information  in  addition  to  the  Easting  and  Northing  Coordinates. 

b.  False  Alarm  Mitigation:  Correlation  and  Discriminant  Analysis 

This  section  constitutes  a  discriminant  analysis  for  scenario  3,  similar  to  that 
already  given  for  scenario  1  and  2.  Two  variables  are  not  examined  because  they  are 
highly  correlated  (that  is,  redundant)  with  others:  magnetic  moment  (correlated  with 
radius)  and  nonferrous  size  (correlated  with  ferrous  size).  Table  11-17,  an  initial  survey  of 
the  correlations,  is  constructed  from  the  68  observations  for  which  all  variables  were 
available.1  Scatterplots  are  shown  in  Figures  11-28  through  11-30,  for  MAG,  EMI,  and 
EMI  versus  MAG,  respectively.  The  scale  and  detail  are  more  easily  apprehended  on 
separate  histograms,  shown  in  Figures  11-31  and  II-32.2 


1  Except  for  x-y  position,  variables  were  not  available  for  the  two  declarations  with  NRL  cell 
assignments  and  object  numbers  K12  #1  and  L9  #3.  For  the  H13  #2  declaration,  EMI  variables  were 
available,  although  magnetometer  ones  were  not.  These  three  delcarations  added  to  the  68  noted  in  the 
text  above  gave  the  71  total  common  declarations  for  scenario  3,  as  noted  earlier  in  this  document. 

2  The  bin  widths  and  locations  of  the  histogram  in  Figure  11-31  and  11-32  do  not  coincide  with  those  in 
Figures  11-28  through  11-30.  This  produces  a  small  difference  in  the  column  heights. 
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Table  11-17.  Correlation  of  Scenario  3  Variables.  All  information 
was  available  for  68  declarations. 
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Figure  11-28.  Scenario  3  Correlation  of  Magnetometer  Variables. 

Blue  indicates  False  Alarms.  Red  indicates  Detections 
50-percent  ellipses  are  shown. 
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Figure  11-29.  Scenario  3  Correlation  of  EMI  Variables. 

Blue  =  False  Alarms.  Red  =  Detections. 
50-percent  ellipses  are  shown. 
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Figure  11-30.  Scenario  3  Correlation  of  EMI  Variables  with  Magnetometer  Variables. 

Red  =  Detections.  Blue  =  False  Alarms.  50-percent  ellipses  are  shown. 
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Figure  11-32.  Scenario  3  EMI  Histograms 

Statistical  analysis  produced  the  following  linear  discriminant: 

Factor  =  0.512  -  43.691  RMAG  +  0.025  GOFJEM  .  (11-11) 


Figure  n-30  shows  a  histogram  of  detections  and  false  alarms  as  a  function  of  this 
discriminant.  Using  a  threshold  of  -1  would  have  given  a  FAR  reduction  of  32  percent 
(13/41)  for  the  41  false  alarms  for  which  all  information  was  available.  Figure  11-34 
shows  the  interplay  of  the  magnetometer-determined  radius  of  ordnance  (RMAG)  and  the 
EMI  GoF  (GOF_EM).  The  main  contribution  is  from  the  magnetometer  radius.  Indeed,  a 
threshold  on  RMAG  of  just  above  0.05  meters  reduces  the  FAR  by  41  percent.  Further 
investigation  of  these  declarations  (EMI  GoF  =16  percent  for  NRL  declaration  K9  #7 
and  EMI  GoF  =  0  percent  for  NRL  declaration  K9  #8)  is  merited,  but  is  deferred  to  future 
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work.  Without  understanding  these  points  better,  we  must  conclude  that  RMAG  alone  is 
the  better  discriminant.  A  histogram  for  RMAG  is  shown  in  Figure  11-35. 

Two  declarations  at  larger  radii  (0.18  m  and  0.37  m)  in  Figure  II-31(b)  do  not 
appear  in  Figures  11-33,  11-34,  and  n-35  because  they  were  from  magnetometer-only 
declarations:  no  EMI  GoF  was  available  for  them.  Incorporating  the  additional  43 
declarations  with  only  RMAG  data  (that  is,  without  EMI  GoF  data)  gives  a  33-percent 
(28/84)  FAR  reduction,  as  shown  in  Figure  11-36. 
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Figure  11-33.  Scenario  3  Discriminant  for  Common  Declarations. 

Factor  is  defined  in  Eq.  11-1 1 . 
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RMAG 

Figure  11-34.  Scenario  3  Detections  and  False  Alarms  as  a  Function  of  RMAG  and  EMI  GoF. 

EMI  GoF  outliers  are  labeled  by  their  NRL-assigned  cell  and  object  numbers.  Red  =  Detection. 
Blue  Crosses  =  False  Alarms.  50-percent  ellipses  are  shown. 


RMAG 

Figure  35.  Scenario  3  Detections  and  False  Alarms  as  a  Function  of 
RMAG  For  These  Detections 
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There  is  one  further  interesting  item  to  note  regarding  the  use  of  the 
magnetometer-fitted  radius  as  a  criterion  for  mitigating  false  alarms:  one  of  the  missed 
detections  had  a  large  fitted  radius.  Of  the  222  declarations  in  the  third  scenario,  only  7 
were  declared  nonordnance  by  NRL.  One  of  those  was  the  MTADS  declaration  in 
NRL-assigned  cell  L9  and  number  1  (within  the  cell).  This  declaration,  by  both  MAG 
and  EMI,  was  actually  a  155-mm  projectile,  the  only  155-mm  projectile  in  this  scenario. 
This  155-mm  projectile  arguably  may  not  fit  within  the  ordnance  class  of  submunitions 
and  grenades  that  the  operators  were  led  to  expect.  The  other  three  155  projectiles  were 
all  in  the  second  scenario,  artillery  and  mortar. 
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Magnetometer  Fitted  Radius 

Figure  11-36.  Scenario  3  Ordnance  Radius  from  the  Magnetometer  Fit. 

This  is  a  histogram  of  those  declarations  as  ordnance  for  which  there  was  also 
EMI  data.  Red  indicates  detections.  Blue  indicates  false  alarms. 

Returning  to  the  subject  of  FAs,  one  can  make  further  progress  by  following  the 
same  path  as  in  scenario  1.  For  the  common  declarations,  a  nominal  41-percent  FAR 
reduction  was  derived  by  using  a  threshold  on  RMAG.  For  the  declarations  which  had 
only  MAG  data  (that  is,  for  which  no  EMI  data  were  available),  all  the  declarations  were 
false  alarms.  A  possible  mitigation  strategy  is  then  to  simply  discount  all  declarations  (in 
this  scenario)  for  which  one  has  only  MAG  data. 


Doing  so  leaves  the  declarations  with  only  EMI  data.  There  are  101  such  declara¬ 
tions  with  x-y  (Easting-Northing)  coordinates  supplied  from  only  the  EMI.  There  is  one 
more  declaration  that  has  x-y  coordinates  from  both  MAG  and  EMI,  but  has  fit  results 
for  only  EMI.  That  declaration,  3/H13/2,  is  also  incorporated  in  this  discriminant.  The 
statistically  significant  discriminant  turns  out  to  be  the  EMI  goodness-of-fit  (GOF_EM). 
A  histogram  (Figure  11-37)  shows  that  no  significant  FAR  reduction  can  be  expected  in 
this  case. 

Thus,  the  truest  estimate  of  FAR  reduction  is 

^(41x41%  +  2x0%+  49x0%. +43x100%)=  44.3%  .  (11-12) 

The  weights  in  this  expression  arise  from  41  (of  44)  common  false  alarms  for  which  EMI 
and  MAG  variables  are  available,  2  for  which  no  variables  are  available,  49  which  have 
only  EMI  variables  available  (48  EMI  declarations  plus  1  common  declaration),  and  43 
which  have  only  MAG  information  available. 


c 

3 

O 


O 


GOF_EM 


"O 

o 

"D 

O 

3- 

O' 

3 

-o 

CD 

— t 

CD 

Q) 


Figure  11-37.  Scenario  3  EMI  GoF  for  the  EMI-only  Declarations. 

Red  indicates  detections.  Blue  indicates  false  alarms. 
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C.  SENSOR  PERFORMANCE 


In  this  section,  the  MTADS  location  performance  for  detected  ordnance  is 
characterized  by  evaluating  the  detection  parameters  and  determining  trends.  Using  the 
critical  radius  of  1  m,  as  before,  121  of  the  206  ordnance  items  were  detected  by  the 
magnetometer,  while  194  of  206  were  detected  by  EMI.  These  detected  ordnance  are 
examined  in  greater  detail  in  this  section. 

1.  Performance  Dependence  on  Critical  Radius 

PD,  FAR,  and  Pfa  all  depend  on  Rcrit.  The  dependence  of  PD  is  clearest.  As  the  area 
awarded  to  each  detection  is  increased,  PD  increases.  Were  the  distribution  of  ordnance 
random  with  respect  to  the  declarations,  PD  would  scale  as  the  square  of  R^  for  low 
density  of  ordnance.  This  is  not  true  because  the  coordinates  of  declarations  and  ordnance 
are  highly  correlated,  which  is  of  course  the  whole  point  in  using  a  sensor  such  as 
MTADS  and  not  just  drawing  guesses  from  a  hat.  Rather,  declarations  are  either  “close” 
to  ordnance,  say  within  a  meter,  or  are  “far”  from  ordnance,  in  which  case  some  other 
physical  aspect  of  the  site  is  causing  the  UXO-like  signal.  That  is,  there  is  unaccounted- 
for  clutter,  not  “noise”  per  se. 

Figure  11-38  shows  the  dependence  of  PD  on  Rcrll  for  the  EMI  declarations. 
Figure  11-39  shows  the  PD  dependence  for  the  MAG  declarations.  The  EMI  has  generally 
higher  PD  especially  for  larger  Rcril.  Table  11-18  gives  the  mean  and  standard  deviation  of 
the  radial  location  accuracy  and  the  depth  accuracy  (signed  and  absolute  value).  The  EMI 
system  exhibits  the  best  location  accuracy,  but  also  shows  the  largest  depth  error. 

2.  Radial  Location  Accuracy 

For  each  sensor  declaration  that  matched  an  emplaced  ordnance  item,  the 
coordinates  of  the  MTADS  fit  are  recorded.  The  radial  location  error  is  computed:  radial 
location  error  =  fit  location  -  actual  location  of  detected  ordnance. 

a.  Electromagnetic  Induction 

For  the  EMI,  the  mean  radial  location  accuracy  was  0.53  m  with  a  standard 
deviation  of  0.15  m  (see  Table  11-18).  Figure  11-40  shows  the  distribution  of  EMI  radial 
error. 
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Table  11-18.  Detection  Location  Error  and  Depth  Error  for  1-m  Critical  Radius 


Scenario  Sensor 


Error  (m) 


Radial 

Standard 
Deviation 
n)  (m) 


Error  (m) 


Depth 

Standard 
Deviation 
i)  (m) 


Magnitude  Depth 

Standard 
Deviation 
Error  (m)  (m) 


0.14 


Probability  of  Detection  vs.  Critical  Radius 


Figure  11-39.  MAG  Detection  Performance  vs.  Critical  Radius. 

(a)  Scenario  1 ,  Aerial  Gunnery,  (b)  Scenario  2,  Artillery  and  Mortar, 
(c)  Scenario  3,  Submunitions  and  Grenades,  (d)  All  Scenarios. 


Figure  11-40.  EMI  Radial  Error  for  Detected  Ordnance 

Figure  11-41  shows  a  small  dependence  of  GoF  on  radial  error.  The  correlation 
coefficient  is  -0.28,  indicating  a  slight  tendency  for  smaller  radial  error  to  be  associated 
with  better  GoF,  which  is  what  we  expect. 
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b.  Magnetometer 

For  the  magnetometer,  the  mean  radial  location  accuracy  was  0.50  m  with  a 
standard  deviation  of  0.16  m  (see  Table  11-18).  Figure  11-42  shows  the  distribution  of 
MAG  radial  error.  Figure  11-43  shows  some  dependence  of  magnetic  moment  on  radial 
error.  The  correlation  coefficient  of  the  logarithm  of  the  magnetic  moment  (that  is,  as 
plotted)  with  the  radial  error  is  -0.227.  There  is  a  slight  tendency  for  larger  magnetic 
moments  to  be  associated  with  smaller  radial  errors.  Figure  11-44  shows  GoF  as  a 
function  of  radial  error.  The  negative  correlation  coefficient  is  -0.16,  indicating  a  very 
slight  tendency  for  large  GoFs  to  be  associated  with  small  radial  errors.  Figure  11-45 
shows  GoF  as  a  function  of  magnetic  moment.  The  correlation  coefficient  of  0.27 
indicates  a  slight  tendency  of  larger  magnetic  moments  to  be  associated  with  better  fits. 
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Figure  11-41.  EMI  GoF  vs.  Radial  Error 
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3.  Depth  Accuracy 

For  each  sensor  declaration  that  matched  an  emplaced  target  within  the  1-m 
critical  radius,  the  fitted  depth  and  depth  error  (depth  error  =  fitted  depth  -  actual  depth  of 
the  emplaced  target)  were  calculated. 

a.  Electromagnetic  Induction 

For  the  EMI,  the  depth  and  depth  error  are  poorly  correlated  with  the  other 
parameters.  Figure  11-46  shows  the  depth  error  of  detected  ordnance.  The  average  depth 
error  was  7  cm  with  a  standard  deviation  of  0.21  m,  indicating  an  insignificant  bias,  or 
tendency  of  EMI  to  indicate  a  deeper  depth  than  actual.  The  mean  of  the  absolute  value 
of  the  depth  error  was  0.17  m,  with  a  standard  deviation  of  0.14  m.  Figure  11-47  shows 
the  GoF  as  a  function  of  depth.  The  correlation  coefficient  of  0.35  indicates  some 
tendency  of  larger  GoFs  to  go  with  deeper  depths.  Figure  11-48  shows  the  radial  error  as  a 
function  of  depth,  indicating  some  negative  correlation  (correlation  coefficient  of  -0.26). 
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Figure  11-44.  Magnetometer  GoF  vs.  Radial  Error 
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Figure  11-45.  Magnetometer  GoF  vs.  Magnetic  Moment 
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Figure  11-46.  EMI  Depth  Error 


11-48 


Depth  (m) 


Figure  11-47.  EMI  GoF  vs.  Depth 
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Figure  11-48.  EMI  Radial  Error  vs.  Depth 


b.  Magnetometer 

The  average  magnetometer  depth  error  was  0.21  m  with  a  standard  deviation  of 
0.16  m.  These  values  were  the  same  as  the  mean  and  standard  deviation  of  the  absolute 
value  of  the  depth  accuracy  because  so  few  points  were  negative.  This  indicates  a 
significant  tendency  for  the  magnetometer  to  overestimate  the  depth  (by  21  cm).  The 
median  depth  error  was  17  cm.  Figure  11-49  shows  the  magnetometer’s  distribution  of 
depth  error.  Figure  11-50  shows  the  dependence  of  depth  error  on  magnetic  moment,  with 
correlation  coefficient  0.54.  Figure  11-51  shows  the  relationship  of  depth  to  magnetic 
moment,  clearly  indicating  (correlation  coefficient  of  0.72)  that  the  smaller  magnetic 
moment  detections  occur  at  shallower  depths,  as  we  would  expect.  Figure  n-52  shows  the 
GoF  dependence  on  depth,  with  correlation  coefficient  0.30.  Figure  11-53  shows  the 
magnetometer  radial  error  as  a  function  of  depth,  with  correlation  coefficient  -0.21. 
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Figure  11-49.  Magnetometer  Depth  Error 
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4.  Declarations  Common  to  Magnetometer  and  EMI 

We  analyzed  COM  declarations  to  understand  the  general  trends  associated  with 
detection  by  multiple  systems.  This  section  contains  a  comparison  of  the  MTADS-DAS 
fit  parameters  for  this  sensor  pair.  Detections  are  common  when  they  both  match  to  the 
same  piece  of  ordnance.  False  alarms  are  common  when  they  are  within  1.0  m  of  each 
other. 


a.  Common  Detections 

All  declarations  within  a  1.0-m  of  a  baseline  ordnance  item  are  considered  to 
determine  a  common  detection.  Thus,  the  distance  between  common  declarations  can  be 
as  great  as  2.0  m.  The  different  fit  parameters  are  considered  to  establish  trends  that 
might  be  exploitable  for  combining  or  fusing  the  two  data  sets. 

b.  Detection  Radial  Location  Offset 

The  location  offset  is  the  difference  between  the  fitted  location  of  each  detected 
baseline  item  (on  the  plane  of  Earth’s  surface)  for  the  EMI  system  and  the  magnetometer. 

One  hundred  twenty  baseline  items  have  detections  in  common.  The  mean  radial 
distance  between  declarations  for  each  baseline  ordnance  item  detected  by  both  sensors  is 
0.21  m,  with  a  standard  deviation  of  0.16  m.  The  detections  are  highly  clustered  within  an 
area  of  radius  0.5  m  surrounding  the  baseline  item  (see  Figure  11-54).  Figure  11-55 
indicates  that  there  is  some  correlation  (correlation  coefficient  of  0.37)  between  the  EMI 
and  MAG  radial  errors. 


Figure  11-54.  Common  Detection  Radial  Location  Offset 
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Figure  11-55.  EMI  Radial  Error  vs.  Magnetometer  Radial  Error 
for  Common  Detections 

c.  Detection  Depth  Offset 

The  depth  offset  is  the  difference  between  the  fitted  depth  of  each  detected 
baseline  item  (in  the  plane  of  Earth’s  surface)  for  the  two  sensors;  that  is,  the  target  depth 
estimated  from  the  EMI  data  subtracted  from  that  estimated  from  the  magnetometer  data. 
The  depth  offset  can  be  positive  or  negative.  The  trend  of  the  sign  indicates  whether  one 
sensor  consistently  estimates  the  depth  of  the  baseline  item  to  be  greater  than  the  fitted 
depth  from  the  other  sensor. 

Figure  11-56  gives  the  common  detection  depth  offsets.  Note  that  the  depth  offsets 
are  for  locations  already  known  to  be  within  2  x  Rcrit  of  each  other. 

Figure  11-57  shows  the  relationship  between  depth  errors  from  the  two  sensors, 
with  correlation  coefficient  0.37.  Figure  11-58  shows  the  relationship  between  depths 
from  the  two  sensors.  These  have  a  high  correlation  coefficient  of  0.89. 

d.  Detection  Goodness-of-Fit  Comparison 

Each  time  the  MTADS-DAS  attempts  to  fit  a  designated  anomaly  with  the 
appropriate  model,  a  measure  of  the  relative  fit  of  the  model  to  the  actual  data  is 
established.  Because  of  the  fit  methodology,  fits  with  GoF  less  than  0.90  are  quite  poor. 
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Figure  11-56.  EMI-MAG  Depth  Offset  for  Detected  Ordnance 
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Figure  11-57.  EMI  Depth  Error  vs.  Magnetometer  Depth 
Error  for  Detected  Ordnance 

These  poor  fits  are  generally  associated  with  a  very  noisy  background  or  with  atypical 
objects  in  the  ground.  As  shown  in  Figure  11-59,  common  detections  for  the 
magnetometer  and  EMI  exhibit  some  correlation  between  their  GoFs.  The  correlation 
coefficient  is  0.31. 


11-55 


5.  False  Alarms  Common  to  Magnetometer  and  EMI 

This  is  an  effort  to  understand  false  alarms  by  examining  the  declarations  that 
correspond  to  nonordnance  items  in  the  baseline.  The  weak  link  in  this  exercise  is  the 
assumption  that  the  nonordnance  baseline  items  are  representative  of  sources  of  false 
alarms  in  real-world  employment  of  the  MTADS.  We  first  examine  this  assumption. 
Then,  we  present  the  false-alarm  analysis  proper. 

a.  Separation  of  Nonordnance  and  Nonbaseline  Items 

There  are  two  sources  of  false  alarms  in  the  test  of  MTADS  at  JPG:  the 
nonordnance  items  in  the  baseline  and  the  items  that  are  not  in  the  baseline,  called 
“nonbaseline”  items.  We  examine  the  similarity  of  these  two  false-alarm  sources  by 
doing  a  discriminant  analysis  as  done  for  the  scenario-specific  false-alarm  mitigation. 
The  purpose  here  is  different;  we  want  to  gauge  the  validity  of  taking  the  nonordnance 
items  as  representative  of  the  nonbaseline  items  in  the  following  analysis  of  false  alarms. 
We  examine  the  degree  to  which  nonordnance  and  nonbaseline  items  differ  by  doing  a 
Kolmogorov-Smimov  comparison  test  on  the  most  important  discriminant. 

Since  three  classes  of  declarations  as  ordnance  are  used  in  the  discriminant 
analysis,  two  linear  (Fisher)  discriminants,  or  factors,  are  derived  for  each  case  examined. 
The  three  classes  are  ordnance,  nonordnance,  and  nonbaseline. 

Figures  11-60  through  11-63  show  the  graphical  results  of  the  discriminant 
analysis,  together  with  definitions  of  the  two  discriminants,  FACTOR(l)  and 
FACTOR(2).  Table  11-19,  the  jackknifed  percent  of  correct  classification  by  the  resultant 
discriminants,  gives  an  indication  of  how  good  the  discriminants  are.  Table  11-20  shows 
how  much  of  the  dispersion  in  Figures  11-60  through  11-63  can  be  accounted  for  by  the 
factors.  By  construction  FACTOR(l)  is  the  dominant  factor,  that  is,  it  accounts  for  the 
most  dispersion.  Distributional  characteristics  for  FACTOR(l)  for  declarations 
corresponding  to  ordnance,  false  alarm  (just  nonbaseline  plus  nonordnance),  nonbaseline, 
and  nonordnance  items  are  also  given  in  Table  11-20. 

Table  11-21  shows  the  results  of  the  Kolmogorov-Smimov  two-sample  test. 
Notice  that  the  nonordnance  and  nonbaseline  classes  differ  strongly  (significance  level  of 
0.001  or  less).  This  makes  the  exercise  of  using  one  as  a  surrogate  for  the  other  suspect, 
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FACTOR(I) 

Figure  11-60.  Scenario  1  Ordnance,  Nonordnance,  and  Nonbaseline  Distributions 
as  Functions  of  Factors  from  a  Linear  (Fisher)  Discriminant  Analysis. 

FACTOR(I)  =  -13.513  +  7.751  RFE_EM  +  0.113  GOF_MAG  +  0.025  GOF_EM. 
FACTOR(2)  =  10.128  +  12.225  RFE_EM  -  0.108  GOF_MAG  -  0.020  GOF_EM. 
50-percent  ellipses  are  shown. 
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FACTOR(2) 


FACT0R(1 ) 

Figure  11-61.  Scenario  2  Ordnance,  Nonordnance,  and  Nonbaseline  Distributions 
as  Functions  of  Factors  from  a  Linear  (Fisher)  Discriminant  Analysis. 

FACTOR(I)  =  -6.605  +  0.482  ZMAG  +  0.009  INCLIN  +  0.076  GOF_EM. 
FACTOR(2)  =  -0.892  +  1.636  ZMAG  +  0.017  INCLIN  -  0.011  GOF_EM. 
50-percent  ellipses  are  shown. 
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FACTOR(2) 


FACT0R(1 ) 


Figure  11-62.  Scenario  3  Ordnance,  Nonordnance,  and  Nonbaseline  Distributions 
as  Functions  of  Factors  from  a  Linear  (Fisher)  Discriminant  Analysis. 

FACTOR(I)  =  1.919  +  6.748  RMAG  -  66.979  RFE_EM. 

FACTOR(2)  =  -1.276  +  50.503  RMAG  -  24.513  RFE_EM. 

50-percent  ellipses  are  shown. 
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Figure  11-63.  Combined  Scenarios’  Ordnance,  Nonordnance,  and  Nonbaseline  Distributions 
as  Functions  of  Factors  from  a  Linear  (Fisher)  Discriminant  Analysis. 

FACTOR(I)  =  -3.802  +  0.006  INCLIN  +  0.049  GOF_EM. 

FACTOR (2)  =  -0.459  +  0.026  INCLIN  -  0.010  GOF_EM 
50-percent  ellipses  are  shown. 


Table  11-19.  Jackknifed  Correct  Classification  by  Linear  Discriminants 


Scenario 

Percent  Correct 

Ordnance 

Nonordnance 

Nonbaseline 

All 

1 

50 

68 

67 

62 

2 

78 

72 

78 

76 

3 

63 

69 

72 

68 

All 

62 

54 

72 

61 

at  least  in  so  far  as  the  variables  making  up  the  discriminants  are  concerned.  It  is  thus,  on 
the  variables  that  do  not  figure  in  the  discriminants,  shown  in  Figures  11-60  through  11-63, 
given  again  in  Table  11-22  for  emphasis,  that  the  following  analysis  is  more  persuasive. 
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Table  11-20.  Separation  Statistics  for  the  Dominant  Discriminant,  FACTOR(I). 

The  definitions  of  FACTOR(I)  and  FACTOR(2)  are  different  for  each  scenario. 


Scenario 

Dispersion  [%] 

Values  of  FACTOR(I) 
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59 

-0.1 

0.8 

12 

-1.8 

1.7 

m 

8.3 

65 

1 

0.6 

62 

-1.1 

1.8 

39 

-0.1 

0.9 

23 

-2.7 

1.8 

m 

64.9 

35.1 

27 

-0.1 

0.7 

41 

0.1 

1.5 

16 

-1.1 

1.6 

25 

0.8 

0.7 

All 

88.6 

11.4 

124 

0.6 

0.9 

174 

-0.4 

1.3 

114 

0.2 

1 

60 

-1.5 

1.2 

Table  11-21.  Kolmogorov-Smirnov  Two-Sample  Test  for  Ordnance,  Nonordnance,  and 
Nonbaseline  Items.  The  two  samples  indicated  are  tested  as  a  function  of  FACTOR(1 ). 
The  effective  number  of  data  is  Ne  =  N,xN2  /  (N,+  Na),  where  N,  is  the  number  of  data 
from  the  first  sample  and  N2  is  the  number  of  data  from  the  second  sample. 


Scenario 

Ordnance  -  False 
Alarms 
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22.1 

0.419 

0.001 

20.7 

0.391 

0.003 

8.7 

0.722 

<0.001 

10 

0.594 

0.001 

m 

31.7 

0.665 

<0.001 

24.4 

0.581 

<0.001 

17 

0.873 

<0.001 

14.5 

0.71 

<0.001 

m 

16.3 

0.422 

0.004 

10 

0.48 

0.012 

13 

0.616 

<0.001 

9.8 

0.623 

0.001 

EM 

72.4 

0.415 

<0.001 

59.4 

0.299 

<0.001 

40.4 

0.718 

<0.001 

39.3 

0.619 

<0.001 

Table  11-22.  Definitions  of  FACTOR(1)s  for  Three-Way  Discrimination 


Scenario 

Definition  of  FACTOR(I) 

1 

-13.513  +  7.751  RFE_EM  +  0.113  GOF_MAG  +  0.025  GOF_EM 

2 

-6.605  +  0.482  ZMAG  +  0.009  INCLIN  +  0.076  GOF_EM 

3 

1 .91 9  +  6.748  RMAG  -  66.979  RFE_EM 

All 

-3.802  +  0.006  INCLIN  +  0.049  GOF_EM 
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b.  Radial  Location  Accuracy 

An  analysis  of  the  radial  location  accuracy  of  the  false  alarms  (detected  non¬ 
ordnance  items  present  in  the  baseline)  reveals  that  both  sensors  localize  most  of  the 
nonordnance  items  within  0.5-m  halo.  In  the  submunitions  and  grenades  scenario,  the 
nonordnance  are  emplaced  at  more  shallow  depths,  where  nonanthropic  clutter  is  most 
dense.  This  causes  the  mean  radial  error  for  this  scenario  to  increase  10  cm  for  both  the 
magnetometer  and  the  EMI.  However,  since  the  nonordnance  baseline  items  were 
composed  of  metal,  it  is  not  surprising  that  the  fit  confidences  were  quite  high  in  all 
scenarios. 


Table  11-23.  Radial  Location  Accuracy 


Sensor 

Scenario 

FAR 

Mean 
Radial 
Error  (m) 

Standard 

Deviation 

Fit 

Standard 

Deviation 

EMI 

Aerial 

Gunnery 

0.62 

0.53 

0.17 

80 

17 

EMI 

Artillery  and 
Mortar 

0.36 

0.48 

0.14 

83 

10 

EMI 

Submuni¬ 
tions  and 
Grenade 

0.39 

0.61 

0.23 

54 

33 

Magnetometer 

Aerial 

Gunnery 

0.57 

0.50 

0.17 

94 

4 

Magnetometer 

Artillery  and 
Mortar 

0.40 

0.48 

0.10 

92 

6 

Magnetometer 

Submuni¬ 
tions  and 
Grenade 

0.21 

0.63 

0.10 

82 

21 

c.  Depth  Location  Accuracy 

As  Table  11-24  shows,  the  mean  depth  error  for  the  EMI  and  magnetometer  ranges 
from  18  to  34  cm  and  from  9  to  18  cm,  respectively.  In  each  scenario,  the  magnetometer 
outperforms  the  EMI  sensor  in  depth  estimation  by  approximately  9  cm  or  more.  This  is 
most  likely  due  to  the  MTADS  fitting  algorithm  for  the  EMI  sensor,  which  does  not 
provide  precision  depth  fittings.  The  depth  error  for  these  near-surface,  metallic,  non¬ 
ordnance  items  is  in  accordance  with  previous  analyses  of  MTADS  system  performance. 
Individually,  the  MTADS  fitting  parameters  are  not  useful  to  discriminate  between 
nonordnance  and  ordnance  in  these  scenarios. 
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Table  11-24.  Depth  Accuracy 


Sensor 

Scenario 

FAR 

Mean 
Depth 
Error  (m) 

Standard 

Deviation 

Standard 

Deviation 

EMI 

Aerial 

Gunnery 

0.62 

0.27 

0.21 

0.60 

0.46 

EMI 

Artillery  and 
Mortar 

0.36 

0.34 

0.25 

0.50 

0.28 

EMI 

Submuni¬ 
tions  and 
Grenade 

0.39 

0.18 

0.11 

0.18 

0.23 

Magnetometer 

Aerial 

Gunnery 

0.57 

HB 

0.12 

0.70 

0.36 

Magnetometer 

Artillery  and 
Mortar 

0.4 

m 

0.14 

Magnetometer 

Submuni¬ 
tions  and 
Grenade 

0.21 

0.09 

0.1 

0.26 

0.09 
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III.  CONCLUSIONS 


This  document  analyzed  the  results  of  a  test  of  MTADS  at  Jefferson  Proving 
Ground  (JPG)  from  14  to  24  January  1997.  The  probability  of  detection,  PD,  is  shown  in 
Table  ffl-1. 


Table  111-1 .  Detection 


Scenario 

Number  of 
Baseline 
Ordnance 

Number  of 
Ordnance 
Declarations 

Number  of  Correct 
Ordnance 
Declarations 

pD(%) 

1 — Aerial  Gunnery 

47 

185 

45 

95.7 

2 — Artillery  and  Mortars 

73 

216 

70 

95.9 

3 — Submunitions  and 
Grenades 

86 

215 

80 

93 

The  number  of  false  alarms  (FAs)  is  defined  as  the  number  of  declarations  as 
ordnance  that  did  not  correspond  to  baseline  ordnance.  The  false  alarm  rate  is  shown  in 
Table  III-2. 


Table  111-2.  False  Alarm  Rates 


Scenario 

Area  Surveyed 
(Square  Meters) 

Number  of  False 
Alarms 

False  Alarm  Rate 
(per  Hectare)* 

1 — Aerial  Gunnery 

33,445  (8.265  acres) 

140 

41.9 

2 — Artillery  and  Mortars 

39,391  (9.735  acres 

146 

37.6 

3 — Submunitions  and 
Grenades 

29,729  (7.347  acres) 

135 

45.4 

*  A  hectare  is  1 0,000  square  meters  or  2.47  acres. 


The  addition  of  EMI  sensor  information  was  crucial  for  reducing  the  FAR, 
particularly  in  scenarios  1  and  3,  where  it  took  part  in  every  good  detection.  In  scenario  1 
it  was  not,  by  itself,  responsible  for  any  false  alarms.1  Table  III-3  shows  the  discriminants 
that  were  derived  in  each  case,  and  the  potential  FAR  reduction. 


1  This  assumes  that  those  declarations  for  which  there  were  both  EMI  and  MAG  data  were  not  more 
difficult  (in  some  sense)  than  those  for  which  only  MAG  or  only  EMI  data  were  available.  That  is,  no 
essential  synergy  is  assumed  between  MAG  and  EMI. 
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Table  111-3.  Potential  FAR  Reduction  by  Using  Linear  Discriminants 


Scenario 

Declaration 

Type 

Proportion  of 
FAs 

Discriminant 

Possible  FAR 
Reduction 

Total  FAR 
Reduction 

1 

Common  and 
EMI-only 

58%  (81/140) 

RFE_EM 

16% 

51.4% 

MAG-only 

42%  (59/140) 

Not  Applicable 

100% 

2 

Common 

42.5% 

(62/146) 

Function  of 
ZMAG, 

INCLIN, 

GOF_EM 

60% 

25.5% 

MAG-only 

45.2  (66/146) 

None 

0% 

EMI-only 

12.3% 

(18/146) 

None 

0% 

3 

Common 
(excluding  3) 

30.4% 

(41/135) 

RMAG 

41% 

44% 

MAG-only 

31.9% 

(49/135) 

Not  Applicable 

100% 

EMI-only 
(including  1 
common) 

36.3  (49/135) 

None 

0% 

No  Information 
(excluded  from 
common) 

1.5%  (2/135) 

None 

0% 

The  obvious  implications  are  that  between  25-  and  50-percent  reduction  in  false 
alarms  should  be  achievable  with  fairly  rudimentary  discrimination  algorithms.  An 
important  caveat  to  this  finding  is  that  it  is  essential  to  know  what  kind  of  scenario  one  is 
in.  No  useful  discriminant  was  found  that  was  good  in  all  three  scenarios.  Also,  it  should 
be  emphasized  that  Table  III-3  accurately  shows  the  FAR  reduction  that  would  have  been 
achieved  in  the  JPG  MTADS  test,  it  is  not  necessarily  indicative  of  performance  in  other 
tests.  Specifically,  the  discriminant  thresholds  chosen  were  such  as  to  allow  zero  missed 
detections  on  account  of  the  false  alarm  mitigation.  The  real  population  distribution  may 
have  tails  that  would  imply  some  number  of  missed  detections  as  the  tradeoff  for 
reducing  the  FAR. 

Tests  and  surveys  are  needed  to  determine  real  population  distributions  for  UXO. 
The  implications  of  these  distributions  for  realistic  false  alarm  mitigation  also  depends  on 
better  understanding  of  the  detection  physics.  These  areas  need  to  be  further  studied  if  the 
indicated  false  alarms  mitigation  is  to  be  achieved  in  practice.  Indeed,  since  the  statistical 
discrimination  approach  taken  in  this  report  is  essentially  conservative  (using  only 
statistically  significant  factors,  with  only  linear  discriminants),  there  is  reason  to  believe 
physics-based  approaches  could  do  better  than  the  analysis  of  Table  III-3  indicates. 
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